Exploring the Latent Capacity of LLMs for One-Step Text Generation
A recent study showed that large language models (LLMs) can reconstruct
surprisingly long texts - up to thousands of tokens - via autoregressive
generation from just one specially trained input embedding. In this work, we
explore whether such reconstruction is possible without autoregression. We show
that frozen LLMs can generate hundreds of accurate tokens in just one forward
pass, when provided with only two learned embeddings. This reveals a surprising
and underexplored capability of LLMs - multi-token generation without iterative
decoding. We investigate the behaviour of these embeddings and provide insight
into the type of information they encode. We also empirically show that
although these representations are not unique for a given text, they form
connected and local regions in embedding space - a property that suggests the
potential of learning a dedicated encoder into that space.
6836babd75a4c5486bac4149
Gleb Mezentsev
extracted_confirmed
2025-05-28T07:43:05.521Z
672e0638ee49faac3ad53af7
/avatars/3a273b4beba8286309296a1e25bc34a9.svg
Gleb Mezentsev
user
glebzok
6836babd75a4c5486bac414a
Ivan Oseledets
admin_assigned
2025-05-28T12:15:02.162Z
6169a581d05945bfd8718dfa
/avatars/1892ab06a7ddb557232777de3cbec470.svg
Ivan Oseledets
user
oseledets
Exploring the Latent Capacity of LLMs for One-Step Text Generation
A recent study showed that large language models (LLMs) can reconstruct
surprisingly long texts - up to thousands of tokens - via autoregressive
generation from just one specially trained input embedding. In this work, we
explore whether such reconstruction is possible without autoregression. We show
that frozen LLMs can generate hundreds of accurate tokens in just one forward
pass, when provided with only two learned embeddings. This reveals a surprising
and underexplored capability of LLMs - multi-token generation without iterative
decoding. We investigate the behaviour of these embeddings and provide insight
into the type of information they encode. We also empirically show that
although these representations are not unique for a given text, they form
connected and local regions in embedding space - a property that suggests the
potential of learning a dedicated encoder into that space.
LLMs can generate long text segments in a single forward pass using learned embeddings, revealing a capability for multi-token generation without iterative decoding.
large language models autoregressive generation input embedding frozen LLMs multi-token generation iterative decoding learned embeddings embedding space dedicated encoder