SIM-CoT: Supervised Implicit Chain-of-Thought
Implicit Chain-of-Thought (CoT) methods present a promising, token-efficient
alternative to explicit CoT reasoning in Large Language Models (LLMs), but a
persistent performance gap has limited the application of implicit CoT. We
identify a core latent instability issue by scaling the computational budget of
implicit CoT approaches: as we increase the number of implicit reasoning tokens
to enhance performance, the training process often becomes unstable and
collapses. Our analysis reveals that this instability arises from the latent
representations becoming homogeneous and losing their semantic diversity, a
failure caused by insufficient step-level supervision in existing implicit CoT
approaches. To address this issue, we propose SIM-CoT, a plug-and-play training
module that introduces step-level supervision to stabilize and enrich the
latent reasoning space. Specifically, SIM-CoT employs an auxiliary decoder
during training to align each implicit token with its corresponding explicit
reasoning step, ensuring that latent states capture distinct and meaningful
information. The proposed auxiliary decoder is removed during inference,
preserving the computational efficiency of implicit CoT methods with no added
overhead. In addition, the auxiliary decoder affords interpretability of
implicit reasoning by projecting each latent token onto an explicit reasoning
vocabulary, enabling per-step visualization of semantic roles and diagnosis.
SIM-CoT significantly enhances both the in-domain accuracy and out-of-domain
stability of various implicit CoT methods, boosting baselines like Coconut by
+8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. Demonstrating strong
scalability, SIM-CoT also surpasses the explicit CoT baseline on GPT-2 by 2.1%
with 2.3\times greater token efficiency, while substantially closing the
performance gap on larger models like LLaMA-3.1 8B.
68d4b42136950a9dff1568d1
Xilin Wei
claimed_verified
2025-09-25T11:35:19.953Z
62eb70462f0f5e54df42f778
/avatars/456049dba67638d3cdb330cdf383f272.svg
Xilin Wei
user
Wiselnn
68d4b42136950a9dff1568d2
Xiaoran Liu
claimed_verified
2025-09-25T07:10:49.956Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Xiaoran Liu
user
SII-xrliu
68d4b42136950a9dff1568d3
Yuhang Zang
claimed_verified
2025-09-26T12:20:55.208Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
68d4b42136950a9dff1568d4
Xiaoyi Dong
68d4b42136950a9dff1568d5
Yuhang Cao
68d4b42136950a9dff1568d6
Jiaqi Wang
68d4b42136950a9dff1568d7
Xipeng Qiu
68d4b42136950a9dff1568d8
Dahua Lin
SIM-CoT: Supervised Implicit Chain-of-Thought
Implicit Chain-of-Thought (CoT) methods present a promising, token-efficient
alternative to explicit CoT reasoning in Large Language Models (LLMs), but a
persistent performance gap has limited the application of implicit CoT. We
identify a core latent instability issue by scaling the computational budget of
implicit CoT approaches: as we increase the number of implicit reasoning tokens
to enhance performance, the training process often becomes unstable and
collapses. Our analysis reveals that this instability arises from the latent
representations becoming homogeneous and losing their semantic diversity, a
failure caused by insufficient step-level supervision in existing implicit CoT
approaches. To address this issue, we propose SIM-CoT, a plug-and-play training
module that introduces step-level supervision to stabilize and enrich the
latent reasoning space. Specifically, SIM-CoT employs an auxiliary decoder
during training to align each implicit token with its corresponding explicit
reasoning step, ensuring that latent states capture distinct and meaningful
information. The proposed auxiliary decoder is removed during inference,
preserving the computational efficiency of implicit CoT methods with no added
overhead. In addition, the auxiliary decoder affords interpretability of
implicit reasoning by projecting each latent token onto an explicit reasoning
vocabulary, enabling per-step visualization of semantic roles and diagnosis.
SIM-CoT significantly enhances both the in-domain accuracy and out-of-domain
stability of various implicit CoT methods, boosting baselines like Coconut by
+8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. Demonstrating strong
scalability, SIM-CoT also surpasses the explicit CoT baseline on GPT-2 by 2.1%
with 2.3\times greater token efficiency, while substantially closing the
performance gap on larger models like LLaMA-3.1 8B.
SIM-CoT, a plug-and-play training module, introduces step-level supervision to stabilize and enrich the latent reasoning space of implicit Chain-of-Thought methods, enhancing their performance and efficiency.
implicit Chain-of-Thought explicit Chain-of-Thought Large Language Models latent instability step-level supervision SIM-CoT auxiliary decoder latent representations semantic diversity in-domain accuracy out-of-domain stability Coconut CODI GPT-2 LLaMA-3.1 8B token efficiency