Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
  Feedback
Large language models (LLMs) have recently demonstrated remarkable success in
mathematical reasoning. Despite progress in methods like chain-of-thought
prompting and self-consistency sampling, these advances often focus on final
correctness without ensuring that the underlying reasoning process is coherent
and reliable. This paper introduces Step-KTO, a training framework that
combines process-level and outcome-level binary feedback to guide LLMs toward
more trustworthy reasoning trajectories. By providing binary evaluations for
both the intermediate reasoning steps and the final answer, Step-KTO encourages
the model to adhere to logical progressions rather than relying on superficial
shortcuts. Our experiments on challenging mathematical benchmarks show that
Step-KTO significantly improves both final answer accuracy and the quality of
intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
achieves a notable improvement in Pass@1 accuracy over strong baselines. These
results highlight the promise of integrating stepwise process feedback into LLM
training, paving the way toward more interpretable and dependable reasoning
capabilities.
679208664e521de952ca0cdc
Yen-Ting Lin
claimed_verified
2025-01-23T09:17:30.742Z
5df9c78eda6d0311fd3d541f
https://cdn-avatars.huggingface.co/v1/production/uploads/5df9c78eda6d0311fd3d541f/8oDFuP77l5zhvamXNVmnc.jpeg
Yen-Ting Lin
user
yentinglin
679208664e521de952ca0cdd
Di Jin
admin_assigned
2025-01-24T10:15:36.716Z
62f690dfc58915315c504af5
/avatars/a6732dda8cd7e37d9c0e0b1dfb68c66b.svg
Di Jin
user
jindi
679208664e521de952ca0cde
Tengyu Xu
679208664e521de952ca0cdf
Tianhao Wu
679208664e521de952ca0ce0
Sainbayar Sukhbaatar
admin_assigned
2025-01-24T10:16:23.763Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
679208664e521de952ca0ce1
Chen Zhu
679208664e521de952ca0ce2
Yun He
admin_assigned
2025-01-24T10:16:34.273Z
6437de5d51c7ebfc813ce68a
/avatars/144cb1c5d5a4a645080611953494f437.svg
he
user
yunhe
679208664e521de952ca0ce3
Yun-Nung Chen
679208664e521de952ca0ce4
Jason Weston
admin_assigned
2025-01-24T10:15:25.564Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
679208664e521de952ca0ce5
Yuandong Tian
admin_assigned
2025-01-24T10:16:47.602Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
679208664e521de952ca0ce6
Arash Rahnama
679208664e521de952ca0ce7
Sinong Wang
admin_assigned
2025-01-24T10:17:03.211Z
65b483c5ed110eb9f1ee62df
/avatars/29100098f5aed1735675d06c516a85b7.svg
Sinong Wang
user
TheronWong
679208664e521de952ca0ce8
Hao Ma
679208664e521de952ca0ce9
Han Fang
Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
  Feedback
Large language models (LLMs) have recently demonstrated remarkable success in
mathematical reasoning. Despite progress in methods like chain-of-thought
prompting and self-consistency sampling, these advances often focus on final
correctness without ensuring that the underlying reasoning process is coherent
and reliable. This paper introduces Step-KTO, a training framework that
combines process-level and outcome-level binary feedback to guide LLMs toward
more trustworthy reasoning trajectories. By providing binary evaluations for
both the intermediate reasoning steps and the final answer, Step-KTO encourages
the model to adhere to logical progressions rather than relying on superficial
shortcuts. Our experiments on challenging mathematical benchmarks show that
Step-KTO significantly improves both final answer accuracy and the quality of
intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
achieves a notable improvement in Pass@1 accuracy over strong baselines. These
results highlight the promise of integrating stepwise process feedback into LLM
training, paving the way toward more interpretable and dependable reasoning
capabilities.
large language models (LLMs) chain-of-thought prompting self-consistency sampling Step-KTO training framework process-level feedback outcome-level feedback logical progressions MATH-500 dataset Pass@1 accuracy