SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings
  and Speaks in Tokens
The recently proposed Large Concept Model (LCM) generates text by predicting
a sequence of sentence-level embeddings and training with either mean-squared
error or diffusion objectives. We present SONAR-LLM, a decoder-only transformer
that "thinks" in the same continuous SONAR embedding space, yet is supervised
through token-level cross-entropy propagated via the frozen SONAR decoder. This
hybrid objective retains the semantic abstraction of LCM while eliminating its
diffusion sampler and restoring a likelihood-based training signal. Across
model sizes from 39M to 1.3B parameters, SONAR-LLM attains competitive
generation quality. We report scaling trends, ablations, benchmark results, and
release the complete training code and all pretrained checkpoints to foster
reproducibility and future research.
689af884fab6fdd2e52ac634
Nikita Dragunov
689af884fab6fdd2e52ac635
Temurbek Rahmatullaev
689af884fab6fdd2e52ac636
Elizaveta Goncharova
689af884fab6fdd2e52ac637
Andrey Kuznetsov
689af884fab6fdd2e52ac638
Anton Razzhigaev
claimed_verified
2025-09-01T07:54:50.398Z
6172aaeec8e66e2aa84c06b9
https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
Anton Razzhigaev
user
razzant
SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings
  and Speaks in Tokens
The recently proposed Large Concept Model (LCM) generates text by predicting
a sequence of sentence-level embeddings and training with either mean-squared
error or diffusion objectives. We present SONAR-LLM, a decoder-only transformer
that "thinks" in the same continuous SONAR embedding space, yet is supervised
through token-level cross-entropy propagated via the frozen SONAR decoder. This
hybrid objective retains the semantic abstraction of LCM while eliminating its
diffusion sampler and restoring a likelihood-based training signal. Across
model sizes from 39M to 1.3B parameters, SONAR-LLM attains competitive
generation quality. We report scaling trends, ablations, benchmark results, and
release the complete training code and all pretrained checkpoints to foster
reproducibility and future research.
SONAR-LLM, a decoder-only transformer using token-level cross-entropy in the SONAR embedding space, achieves competitive text generation quality without diffusion sampling.
Large Concept Model LCM SONAR-LLM decoder-only transformer SONAR embedding space token-level cross-entropy diffusion objectives likelihood-based training signal