Hymba: A Hybrid-head Architecture for Small Language Models
We propose Hymba, a family of small language models featuring a hybrid-head
parallel architecture that integrates transformer attention mechanisms with
state space models (SSMs) for enhanced efficiency. Attention heads provide
high-resolution recall, while SSM heads enable efficient context summarization.
Additionally, we introduce learnable meta tokens that are prepended to prompts,
storing critical information and alleviating the "forced-to-attend" burden
associated with attention mechanisms. This model is further optimized by
incorporating cross-layer key-value (KV) sharing and partial sliding window
attention, resulting in a compact cache size. During development, we conducted
a controlled study comparing various architectures under identical settings and
observed significant advantages of our proposed architecture. Notably, Hymba
achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model
surpasses all sub-2B public models in performance and even outperforms
Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size
reduction, and 3.49x throughput.
674003e7188f876068a7f1be
Xin Dong
674003e7188f876068a7f1bf
Yonggan Fu
admin_assigned
2024-11-22T13:13:01.574Z
66011091d2a378a1633b3a83
https://cdn-avatars.huggingface.co/v1/production/uploads/66011091d2a378a1633b3a83/0a46rrZbY_gR8q_I7ce2B.jpeg
Yonggan Fu
user
tilmto
674003e7188f876068a7f1c0
Shizhe Diao
admin_assigned
2024-11-22T13:13:22.128Z
633bd54b00732349209a18fe
/avatars/150aa5b8d9bcca24366402932bf2d049.svg
Shizhe Diao
user
shizhediao
674003e7188f876068a7f1c1
Wonmin Byeon
admin_assigned
2024-11-22T13:13:28.425Z
66bf958296583c59b049085b
/avatars/04df8dd45835b7ea0991e242784e7810.svg
Wonmin Byeon
user
wbyeon
674003e7188f876068a7f1c2
Zijia Chen
admin_assigned
2024-11-22T13:14:30.968Z
65c2f8350aa2d53135b47b34
/avatars/b08e12905077cf441593586a98b63b2e.svg
Zijia Chen
user
zijiac-nvidia
674003e7188f876068a7f1c3
Ameya Sunil Mahabaleshwarkar
admin_assigned
2024-11-22T13:14:36.408Z
659ef64b419de2ea602cf271
/avatars/b5313218c009cf6d3fb8e3a7ac11091a.svg
Ameya Sunil Mahabaleshwarkar
user
ameyasunilm
674003e7188f876068a7f1c4
Shih-Yang Liu
674003e7188f876068a7f1c5
Matthijs Van Keirsbilck
admin_assigned
2024-11-22T13:14:47.245Z
66476d517ab33d141de67f86
/avatars/e0681c0666c9db2060f91db064112fe9.svg
matthijs van keirsbilck
user
matthijsvk
674003e7188f876068a7f1c6
Min-Hung Chen
admin_assigned
2024-11-22T13:14:52.517Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
674003e7188f876068a7f1c7
Yoshi Suhara
admin_assigned
2024-11-22T13:14:57.438Z
5fc181c4ea82dd667bb0ffae
/avatars/311461de0933d7e4a8a222d0e76f3754.svg
Yoshi Suhara
user
suhara
674003e7188f876068a7f1c8
Yingyan Lin
674003e7188f876068a7f1c9
Jan Kautz
674003e7188f876068a7f1ca
Pavlo Molchanov
claimed_verified
2025-04-20T15:04:56.526Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
Hymba: A Hybrid-head Architecture for Small Language Models
We propose Hymba, a family of small language models featuring a hybrid-head
parallel architecture that integrates transformer attention mechanisms with
state space models (SSMs) for enhanced efficiency. Attention heads provide
high-resolution recall, while SSM heads enable efficient context summarization.
Additionally, we introduce learnable meta tokens that are prepended to prompts,
storing critical information and alleviating the "forced-to-attend" burden
associated with attention mechanisms. This model is further optimized by
incorporating cross-layer key-value (KV) sharing and partial sliding window
attention, resulting in a compact cache size. During development, we conducted
a controlled study comparing various architectures under identical settings and
observed significant advantages of our proposed architecture. Notably, Hymba
achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model
surpasses all sub-2B public models in performance and even outperforms
Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size
reduction, and 3.49x throughput.
Hymba, a family of small language models with a hybrid-head architecture combining transformer attention and state space models, achieves state-of-the-art performance with improved efficiency and reduced cache size.
hybrid-head parallel architecture transformer attention mechanisms state space models (SSMs) attention heads meta tokens cross-layer key-value (KV) sharing partial sliding window attention