Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM's
  Reasoning Capability
Large Language Models (LLMs) have exhibited remarkable performance on
reasoning tasks. They utilize autoregressive token generation to construct
reasoning trajectories, enabling the development of a coherent chain of
thought. In this work, we explore the impact of individual tokens on the final
outcomes of reasoning tasks. We identify the existence of ``critical tokens''
that lead to incorrect reasoning trajectories in LLMs. Specifically, we find
that LLMs tend to produce positive outcomes when forced to decode other tokens
instead of critical tokens. Motivated by this observation, we propose a novel
approach - cDPO - designed to automatically recognize and conduct token-level
rewards for the critical tokens during the alignment process. Specifically, we
develop a contrastive estimation approach to automatically identify critical
tokens. It is achieved by comparing the generation likelihood of positive and
negative models. To achieve this, we separately fine-tune the positive and
negative models on various reasoning trajectories, consequently, they are
capable of identifying identify critical tokens within incorrect trajectories
that contribute to erroneous outcomes. Moreover, to further align the model
with the critical token information during the alignment process, we extend the
conventional DPO algorithms to token-level DPO and utilize the differential
likelihood from the aforementioned positive and negative model as important
weight for token-level DPO learning.Experimental results on GSM8K and MATH500
benchmarks with two-widely used models Llama-3 (8B and 70B) and deepseek-math
(7B) demonstrate the effectiveness of the propsoed approach cDPO.
674d9e3c0912639dc496ae7e
Zicheng Lin
claimed_verified
2025-01-13T09:09:54.784Z
64292eb375bcc24c5e52c011
/avatars/c8cb03ca35ca12d8831be5f4e8547d54.svg
czl
user
Lin1557
674d9e3c0912639dc496ae7f
Tian Liang
674d9e3c0912639dc496ae80
Jiahao Xu
admin_assigned
2024-12-04T11:55:17.273Z
660399710f1fc2f16de18072
/avatars/c22a749cc45db693c2d9ea877c7cace4.svg
Jiahao Xu
user
Jiahao004
674d9e3c0912639dc496ae81
Xing Wang
674d9e3c0912639dc496ae82
Ruilin Luo
claimed_verified
2025-01-12T19:57:51.228Z
6548956fe49bd8d58e8adf0e
/avatars/24c54b14c253c87d4b7438193f16ce28.svg
Ruilin
user
Antimage01
674d9e3c0912639dc496ae83
Chufan Shi
admin_assigned
2024-12-04T11:54:37.406Z
6373b559c3b104ef179cf503
/avatars/36f36eea72a100ee56b37c6d185b47c0.svg
Shi
user
Chufan
674d9e3c0912639dc496ae84
Siheng Li
admin_assigned
2024-12-04T11:54:27.900Z
62f71b23d3bdacb7eec43b1f
https://cdn-avatars.huggingface.co/v1/production/uploads/62f71b23d3bdacb7eec43b1f/6RNVG9zug9tutjo5i8T4C.jpeg
SihengLi
user
SihengLi
674d9e3c0912639dc496ae85
Yujiu Yang
admin_assigned
2024-12-04T11:54:21.597Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
674d9e3c0912639dc496ae86
Zhaopeng Tu
admin_assigned
2024-12-04T11:54:15.917Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM's
  Reasoning Capability
Large Language Models (LLMs) have exhibited remarkable performance on
reasoning tasks. They utilize autoregressive token generation to construct
reasoning trajectories, enabling the development of a coherent chain of
thought. In this work, we explore the impact of individual tokens on the final
outcomes of reasoning tasks. We identify the existence of ``critical tokens''
that lead to incorrect reasoning trajectories in LLMs. Specifically, we find
that LLMs tend to produce positive outcomes when forced to decode other tokens
instead of critical tokens. Motivated by this observation, we propose a novel
approach - cDPO - designed to automatically recognize and conduct token-level
rewards for the critical tokens during the alignment process. Specifically, we
develop a contrastive estimation approach to automatically identify critical
tokens. It is achieved by comparing the generation likelihood of positive and
negative models. To achieve this, we separately fine-tune the positive and
negative models on various reasoning trajectories, consequently, they are
capable of identifying identify critical tokens within incorrect trajectories
that contribute to erroneous outcomes. Moreover, to further align the model
with the critical token information during the alignment process, we extend the
conventional DPO algorithms to token-level DPO and utilize the differential
likelihood from the aforementioned positive and negative model as important
weight for token-level DPO learning.Experimental results on GSM8K and MATH500
benchmarks with two-widely used models Llama-3 (8B and 70B) and deepseek-math
(7B) demonstrate the effectiveness of the propsoed approach cDPO.
The proposed cDPO approach identifies and rewards critical tokens to improve the reasoning accuracy of LLMs by utilizing contrastive estimation between positive and negative models.
autoregressive token generation reasoning trajectories critical tokens contrastive estimation token-level rewards cDPO differential likelihood conventional DPO algorithms token-level DPO GSM8K MATH500 Llama-3 deepseek-math