Memory Augmented Language Models through Mixture of Word Experts
Scaling up the number of parameters of language models has proven to be an
effective approach to improve performance. For dense models, increasing model
size proportionally increases the model's computation footprint. In this work,
we seek to aggressively decouple learning capacity and FLOPs through
Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary
based routing functions and experts. Our proposed approach, dubbed Mixture of
Word Experts (MoWE), can be seen as a memory augmented model, where a large set
of word-specific experts play the role of a sparse memory. We demonstrate that
MoWE performs significantly better than the T5 family of models with similar
number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms
regular MoE models on knowledge intensive tasks and has similar performance to
more complex memory augmented approaches that often require to invoke custom
mechanisms to search the sparse memory.
655c1c3971ed242b42686669
Cicero Nogueira dos Santos
655c1c3971ed242b4268666a
James Lee-Thorp
655c1c3971ed242b4268666b
Isaac Noble
655c1c3971ed242b4268666c
Chung-Ching Chang
655c1c3971ed242b4268666d
David Uthus
Memory Augmented Language Models through Mixture of Word Experts
Scaling up the number of parameters of language models has proven to be an
effective approach to improve performance. For dense models, increasing model
size proportionally increases the model's computation footprint. In this work,
we seek to aggressively decouple learning capacity and FLOPs through
Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary
based routing functions and experts. Our proposed approach, dubbed Mixture of
Word Experts (MoWE), can be seen as a memory augmented model, where a large set
of word-specific experts play the role of a sparse memory. We demonstrate that
MoWE performs significantly better than the T5 family of models with similar
number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms
regular MoE models on knowledge intensive tasks and has similar performance to
more complex memory augmented approaches that often require to invoke custom
mechanisms to search the sparse memory.
A Mixture of Word Experts (MoWE) model demonstrates superior performance in NLP tasks compared to T5 models and traditional MoE models, using memory-augmented techniques and knowledge-rich routing functions.
Mixture-of-Experts MoE Mixture of Word Experts MoWE dense models FLOPs language models knowledge-intensive tasks memory augmented model sparse memory