Variational Reasoning for Language Models
We introduce a variational reasoning framework for language models that
treats thinking traces as latent variables and optimizes them through
variational inference. Starting from the evidence lower bound (ELBO), we extend
it to a multi-trace objective for tighter bounds and propose a forward-KL
formulation that stabilizes the training of the variational posterior. We
further show that rejection sampling finetuning and binary-reward RL, including
GRPO, can be interpreted as local forward-KL objectives, where an implicit
weighting by model accuracy naturally arises from the derivation and reveals a
previously unnoticed bias toward easier questions. We empirically validate our
method on the Qwen 2.5 and Qwen 3 model families across a wide range of
reasoning tasks. Overall, our work provides a principled probabilistic
perspective that unifies variational inference with RL-style methods and yields
stable objectives for improving the reasoning ability of language models. Our
code is available at https://github.com/sail-sg/variational-reasoning.
68d9ef7a0177a6054b013ad7
Xiangxin Zhou
claimed_verified
2025-09-29T05:58:24.272Z
66129c7b50350afe76757262
/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
Xiangxin Zhou
user
zhouxiangxin
68d9ef7a0177a6054b013ad8
Zichen Liu
claimed_verified
2025-10-02T03:09:14.082Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
68d9ef7a0177a6054b013ad9
Haonan Wang
68d9ef7a0177a6054b013ada
Chao Du
68d9ef7a0177a6054b013adb
Min Lin
68d9ef7a0177a6054b013adc
Chongxuan Li
admin_assigned
2025-09-29T13:10:55.814Z
64c07b488e2612254361153b
/avatars/ade0f783cc4c2d3e73f402637f595471.svg
chongxuan li
user
zhenxuan00
68d9ef7a0177a6054b013add
Liang Wang
68d9ef7a0177a6054b013ade
Tianyu Pang
claimed_verified
2025-09-30T06:36:29.585Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
Variational Reasoning for Language Models
We introduce a variational reasoning framework for language models that
treats thinking traces as latent variables and optimizes them through
variational inference. Starting from the evidence lower bound (ELBO), we extend
it to a multi-trace objective for tighter bounds and propose a forward-KL
formulation that stabilizes the training of the variational posterior. We
further show that rejection sampling finetuning and binary-reward RL, including
GRPO, can be interpreted as local forward-KL objectives, where an implicit
weighting by model accuracy naturally arises from the derivation and reveals a
previously unnoticed bias toward easier questions. We empirically validate our
method on the Qwen 2.5 and Qwen 3 model families across a wide range of
reasoning tasks. Overall, our work provides a principled probabilistic
perspective that unifies variational inference with RL-style methods and yields
stable objectives for improving the reasoning ability of language models. Our
code is available at https://github.com/sail-sg/variational-reasoning.
A variational reasoning framework treats thinking traces as latent variables, optimizing them through variational inference to improve language model reasoning.
variational reasoning framework latent variables variational inference evidence lower bound (ELBO) multi-trace objective forward-KL formulation rejection sampling finetuning binary-reward RL GRPO implicit weighting model accuracy probabilistic perspective RL-style methods