Brainformers: Trading Simplicity for Efficiency
Transformers are central to recent successes in natural language processing
and computer vision. Transformers have a mostly uniform backbone where layers
alternate between feed-forward and self-attention in order to build a deep
network. Here we investigate this design choice and find that more complex
blocks that have different permutations of layer primitives can be more
efficient. Using this insight, we develop a complex block, named Brainformer,
that consists of a diverse sets of layers such as sparsely gated feed-forward
layers, dense feed-forward layers, attention layers, and various forms of layer
normalization and activation functions. Brainformer consistently outperforms
the state-of-the-art dense and sparse Transformers, in terms of both quality
and efficiency. A Brainformer model with 8 billion activated parameters per
token demonstrates 2x faster training convergence and 5x faster step time
compared to its GLaM counterpart. In downstream task evaluation, Brainformer
also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM
with a similar number of activated parameters. Finally, Brainformer largely
outperforms a Primer dense model derived with NAS with similar computation per
token on fewshot evaluations.
647975a0a68454566356af42
Yanqi Zhou
claimed_verified
2023-06-05T10:07:51.287Z
643c3c0a9f5d314db2dcba18
/avatars/6e6599c558b88c97144008ada1abb5f7.svg
Yanqi Zhou
user
yanqiz
647975a0a68454566356af43
Nan Du
647975a0a68454566356af44
Yanping Huang
647975a0a68454566356af45
Daiyi Peng
647975a0a68454566356af46
Chang Lan
admin_assigned
2023-06-05T12:06:12.854Z
63003b224f0095df258f3015
/avatars/b4c3957df138b2a4fd296e13b131a5f2.svg
Chang Lan
user
changlan
647975a0a68454566356af47
Da Huang
647975a0a68454566356af48
Siamak Shakeri
647975a0a68454566356af49
David So
647975a0a68454566356af4a
Andrew Dai
647975a0a68454566356af4b
Yifeng Lu
admin_assigned
2023-06-05T12:07:17.432Z
6344a8050f69ad8aa61cb663
/avatars/ed637f510d18ccd1f4bdd6861d5c858e.svg
Yifeng Lu
user
yifenglu
647975a0a68454566356af4c
Zhifeng Chen
647975a0a68454566356af4d
Quoc Le
647975a0a68454566356af4e
Claire Cui
647975a0a68454566356af4f
James Laundon
647975a0a68454566356af50
Jeff Dean
Brainformers: Trading Simplicity for Efficiency
Transformers are central to recent successes in natural language processing
and computer vision. Transformers have a mostly uniform backbone where layers
alternate between feed-forward and self-attention in order to build a deep
network. Here we investigate this design choice and find that more complex
blocks that have different permutations of layer primitives can be more
efficient. Using this insight, we develop a complex block, named Brainformer,
that consists of a diverse sets of layers such as sparsely gated feed-forward
layers, dense feed-forward layers, attention layers, and various forms of layer
normalization and activation functions. Brainformer consistently outperforms
the state-of-the-art dense and sparse Transformers, in terms of both quality
and efficiency. A Brainformer model with 8 billion activated parameters per
token demonstrates 2x faster training convergence and 5x faster step time
compared to its GLaM counterpart. In downstream task evaluation, Brainformer
also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM
with a similar number of activated parameters. Finally, Brainformer largely
outperforms a Primer dense model derived with NAS with similar computation per
token on fewshot evaluations.
Brainformer, a complex block architecture in Transformers, enhances efficiency and performance through diverse layer compositions such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and normalization techniques, outperforming state-of-the-art dense and sparse Transformers.
feed-forward self-attention Brainformer sparsely gated feed-forward layers dense feed-forward layers attention layers layer normalization activation functions GLaM SuperGLUE Primer neural architecture search (NAS)