LLM Pretraining with Continuous Concepts
Next token prediction has been the standard training objective used in large
language model pretraining. Representations are learned as a result of
optimizing for token-level perplexity. We propose Continuous Concept Mixing
(CoCoMix), a novel pretraining framework that combines discrete next token
prediction with continuous concepts. Specifically, CoCoMix predicts continuous
concepts learned from a pretrained sparse autoencoder and mixes them into the
model's hidden state by interleaving with token hidden representations. Through
experiments on multiple benchmarks, including language modeling and downstream
reasoning tasks, we show that CoCoMix is more sample efficient and consistently
outperforms standard next token prediction, knowledge distillation and
inserting pause tokens. We find that combining both concept learning and
interleaving in an end-to-end framework is critical to performance gains.
Furthermore, CoCoMix enhances interpretability and steerability by allowing
direct inspection and modification of the predicted concept, offering a
transparent way to guide the model's internal reasoning process.
67ad783da2808b57a3cd3316
Jihoon Tack
admin_assigned
2025-02-14T15:31:46.173Z
65b15a471566a60bda376e02
/avatars/413340cb2d44c765accb5b3fc450ccc4.svg
Jihoon Tack
user
jihoontack
67ad783da2808b57a3cd3317
Jack Lanchantin
admin_assigned
2025-02-14T15:31:36.784Z
65dd30a6ea1a202d3cb469f7
/avatars/14029def026380e6e20822d916b25a72.svg
Jack Lanchantin
user
jcklcn
67ad783da2808b57a3cd3318
Jane Yu
67ad783da2808b57a3cd3319
Andrew Cohen
67ad783da2808b57a3cd331a
Ilia Kulikov
67ad783da2808b57a3cd331b
Janice Lan
67ad783da2808b57a3cd331c
Shibo Hao
67ad783da2808b57a3cd331d
Yuandong Tian
67ad783da2808b57a3cd331e
Jason Weston
67ad783da2808b57a3cd331f
Xian Li
extracted_confirmed
2025-05-22T23:45:54.667Z
659a395421a7431643caedda
/avatars/c1e0bbcedce68fe3b4fe39e0cf01c65c.svg
Xian Li
user
xlxxl
LLM Pretraining with Continuous Concepts
Next token prediction has been the standard training objective used in large
language model pretraining. Representations are learned as a result of
optimizing for token-level perplexity. We propose Continuous Concept Mixing
(CoCoMix), a novel pretraining framework that combines discrete next token
prediction with continuous concepts. Specifically, CoCoMix predicts continuous
concepts learned from a pretrained sparse autoencoder and mixes them into the
model's hidden state by interleaving with token hidden representations. Through
experiments on multiple benchmarks, including language modeling and downstream
reasoning tasks, we show that CoCoMix is more sample efficient and consistently
outperforms standard next token prediction, knowledge distillation and
inserting pause tokens. We find that combining both concept learning and
interleaving in an end-to-end framework is critical to performance gains.
Furthermore, CoCoMix enhances interpretability and steerability by allowing
direct inspection and modification of the predicted concept, offering a
transparent way to guide the model's internal reasoning process.
A novel pretraining framework, Continuous Concept Mixing (CoCoMix), integrates continuous concepts from a sparse autoencoder into language models, improving sample efficiency and interpretability.
Continuous Concept Mixing CoCoMix sparse autoencoder next token prediction knowledge distillation pause tokens concept learning interleaving interpretability steerability