Scaling Embedding Layers in Language Models
We propose SCONE (Scalable, Contextualized,
Offloaded, N-gram Embedding), a method for
extending input embedding layers to enhance language model performance as layer
size scales. To avoid increased decoding costs, SCONE retains the original
vocabulary while introducing embeddings for a set of frequent n-grams. These
embeddings provide contextualized representation for each input token and are
learned with a separate model during training. During inference, they are
precomputed and stored in off-accelerator memory with minimal impact on
inference speed. SCONE enables two new scaling strategies: increasing the
number of cached n-gram embeddings and scaling the model used to learn them,
all while maintaining fixed inference-time FLOPS. We show that scaling both
aspects allows SCONE to outperform a 1.9B parameter baseline across diverse
corpora, while using only half the inference-time FLOPS.
67a1a51e6aa8429da493d0b5
Da Yu
claimed_verified
2025-02-05T10:15:26.901Z
64b785384df206a3ed142dc0
/avatars/501a90b2c80d9b3a2e0d1819a4211f84.svg
Da Yu
user
Jellyfish0538
67a1a51e6aa8429da493d0b6
Edith Cohen
67a1a51e6aa8429da493d0b7
Badih Ghazi
67a1a51e6aa8429da493d0b8
Yangsibo Huang
admin_assigned
2025-02-04T21:04:05.965Z
645949e7ecf89b6a375ff129
/avatars/93f167fe70c43328b95ed597b6dfa51b.svg
Yangsibo Huang
user
yangsibo
67a1a51e6aa8429da493d0b9
Pritish Kamath
admin_assigned
2025-02-04T21:04:12.118Z
64c0d2f8d76592ba8996036c
/avatars/7b7720ad2060ac5c36651fee8d43ba69.svg
Pritish Kamath
user
pritishkamath
67a1a51e6aa8429da493d0ba
Ravi Kumar
67a1a51e6aa8429da493d0bb
Daogao Liu
admin_assigned
2025-02-04T20:58:14.948Z
667a19869f78b2d01bc402b1
/avatars/b7fb00826a62e70d2dae5f978b7366f3.svg
Daogao Liu
user
ShyShowmaker
67a1a51e6aa8429da493d0bc
Chiyuan Zhang
admin_assigned
2025-02-04T20:58:08.569Z
65c65a2ef2d7e1eb92256d1f
/avatars/b02979bc1549a16515c880ce836c3023.svg
Chiyuan Zhang
user
pluspluskid
Scaling Embedding Layers in Language Models
We propose SCONE (Scalable, Contextualized,
Offloaded, N-gram Embedding), a method for
extending input embedding layers to enhance language model performance as layer
size scales. To avoid increased decoding costs, SCONE retains the original
vocabulary while introducing embeddings for a set of frequent n-grams. These
embeddings provide contextualized representation for each input token and are
learned with a separate model during training. During inference, they are
precomputed and stored in off-accelerator memory with minimal impact on
inference speed. SCONE enables two new scaling strategies: increasing the
number of cached n-gram embeddings and scaling the model used to learn them,
all while maintaining fixed inference-time FLOPS. We show that scaling both
aspects allows SCONE to outperform a 1.9B parameter baseline across diverse
corpora, while using only half the inference-time FLOPS.
SCONE extends language model performance by using contextualized n-gram embeddings, precomputed and stored off-accelerator, to minimize inference costs while maintaining fixed FLOPS.
n-gram embeddings contextualized representation off-accelerator memory inference-time FLOPS