Text Generation Beyond Discrete Token Sampling
In standard autoregressive generation, an LLM predicts the next-token
distribution, samples a discrete token, and then discards the distribution,
passing only the sampled token as new input. To preserve this distribution's
rich information, we propose Mixture of Inputs (MoI), a training-free method
for autoregressive generation. After generating a token following the standard
paradigm, we construct a new input that blends the generated discrete token
with the previously discarded token distribution. Specifically, we employ a
Bayesian estimation method that treats the token distribution as the prior, the
sampled token as the observation, and replaces the conventional one-hot vector
with the continuous posterior expectation as the new model input. MoI allows
the model to maintain a richer internal representation throughout the
generation process, resulting in improved text quality and reasoning
capabilities. On mathematical reasoning, code generation, and PhD-level QA
tasks, MoI consistently improves performance across multiple models including
QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional
training and negligible computational overhead.
682ea25b2c417303938ae8fc
Yufan Zhuang
claimed_verified
2025-05-27T07:55:22.441Z
6438ccbb3b46237de3d052e8
/avatars/baa624d417b0b905e82127dc66346478.svg
Yufan Zhuang
user
yzhuang
682ea25b2c417303938ae8fd
Liyuan Liu
682ea25b2c417303938ae8fe
Chandan Singh
682ea25b2c417303938ae8ff
Jingbo Shang
682ea25b2c417303938ae900
Jianfeng Gao
Text Generation Beyond Discrete Token Sampling
In standard autoregressive generation, an LLM predicts the next-token
distribution, samples a discrete token, and then discards the distribution,
passing only the sampled token as new input. To preserve this distribution's
rich information, we propose Mixture of Inputs (MoI), a training-free method
for autoregressive generation. After generating a token following the standard
paradigm, we construct a new input that blends the generated discrete token
with the previously discarded token distribution. Specifically, we employ a
Bayesian estimation method that treats the token distribution as the prior, the
sampled token as the observation, and replaces the conventional one-hot vector
with the continuous posterior expectation as the new model input. MoI allows
the model to maintain a richer internal representation throughout the
generation process, resulting in improved text quality and reasoning
capabilities. On mathematical reasoning, code generation, and PhD-level QA
tasks, MoI consistently improves performance across multiple models including
QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional
training and negligible computational overhead.
Mixture of Inputs (MoI), a training-free method, enhances autoregressive generation by maintaining a richer internal representation, improving text quality and reasoning capabilities in mathematical reasoning, code generation, and PhD-level QA tasks.
Mixture of Inputs (MoI) autoregressive generation token distribution Bayesian estimation posterior expectation