Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
  Feedback
Large language models (LLMs) have recently demonstrated remarkable success in
mathematical reasoning. Despite progress in methods like chain-of-thought
prompting and self-consistency sampling, these advances often focus on final
correctness without ensuring that the underlying reasoning process is coherent
and reliable. This paper introduces Step-KTO, a training framework that
combines process-level and outcome-level binary feedback to guide LLMs toward
more trustworthy reasoning trajectories. By providing binary evaluations for
both the intermediate reasoning steps and the final answer, Step-KTO encourages
the model to adhere to logical progressions rather than relying on superficial
shortcuts. Our experiments on challenging mathematical benchmarks show that
Step-KTO significantly improves both final answer accuracy and the quality of
intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
achieves a notable improvement in Pass@1 accuracy over strong baselines. These
results highlight the promise of integrating stepwise process feedback into LLM
training, paving the way toward more interpretable and dependable reasoning
capabilities.
679208664e521de952ca0cdc
Yen-Ting Lin
claimed_verified
2025-01-23T09:17:30.742Z
5df9c78eda6d0311fd3d541f
https://cdn-avatars.huggingface.co/v1/production/uploads/5df9c78eda6d0311fd3d541f/8oDFuP77l5zhvamXNVmnc.jpeg
Yen-Ting Lin
user
yentinglin
679208664e521de952ca0cdd
Di Jin
admin_assigned
2025-01-24T10:15:36.716Z
62f690dfc58915315c504af5
/avatars/a6732dda8cd7e37d9c0e0b1dfb68c66b.svg
Di Jin
user
jindi
679208664e521de952ca0cde
Tengyu Xu
679208664e521de952ca0cdf
Tianhao Wu
679208664e521de952ca0ce0
Sainbayar Sukhbaatar
admin_assigned
2025-01-24T10:16:23.763Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
679208664e521de952ca0ce1
Chen Zhu
679208664e521de952ca0ce2
Yun He
admin_assigned
2025-01-24T10:16:34.273Z
6437de5d51c7ebfc813ce68a
/avatars/144cb1c5d5a4a645080611953494f437.svg
he
user
yunhe
679208664e521de952ca0ce3
Yun-Nung Chen
679208664e521de952ca0ce4
Jason Weston
admin_assigned
2025-01-24T10:15:25.564Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
679208664e521de952ca0ce5
Yuandong Tian
admin_assigned
2025-01-24T10:16:47.602Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
679208664e521de952ca0ce6
Arash Rahnama
679208664e521de952ca0ce7
Sinong Wang
admin_assigned
2025-01-24T10:17:03.211Z
65b483c5ed110eb9f1ee62df
/avatars/29100098f5aed1735675d06c516a85b7.svg
Theron S. Wang
user
TheronWong
679208664e521de952ca0ce8
Hao Ma
679208664e521de952ca0ce9
Han Fang
Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
  Feedback
Large language models (LLMs) have recently demonstrated remarkable success in
mathematical reasoning. Despite progress in methods like chain-of-thought
prompting and self-consistency sampling, these advances often focus on final
correctness without ensuring that the underlying reasoning process is coherent
and reliable. This paper introduces Step-KTO, a training framework that
combines process-level and outcome-level binary feedback to guide LLMs toward
more trustworthy reasoning trajectories. By providing binary evaluations for
both the intermediate reasoning steps and the final answer, Step-KTO encourages
the model to adhere to logical progressions rather than relying on superficial
shortcuts. Our experiments on challenging mathematical benchmarks show that
Step-KTO significantly improves both final answer accuracy and the quality of
intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
achieves a notable improvement in Pass@1 accuracy over strong baselines. These
results highlight the promise of integrating stepwise process feedback into LLM
training, paving the way toward more interpretable and dependable reasoning
capabilities.
Step-KTO, a training framework providing binary feedback on both intermediate steps and final answers, enhances the reasoning coherence and accuracy of large language models.
chain-of-thought prompting self-consistency sampling Step-KTO process-level feedback outcome-level feedback trustworthiness reasoning trajectories logical progressions MATH-500 Pass@1 accuracy interpretable reasoning dependable reasoning