Building Math Agents with Multi-Turn Iterative Preference Learning
Recent studies have shown that large language models' (LLMs) mathematical
problem-solving capabilities can be enhanced by integrating external tools,
such as code interpreters, and employing multi-turn Chain-of-Thought (CoT)
reasoning. While current methods focus on synthetic data generation and
Supervised Fine-Tuning (SFT), this paper studies the complementary direct
preference learning approach to further improve model performance. However,
existing direct preference learning algorithms are originally designed for the
single-turn chat task, and do not fully address the complexities of multi-turn
reasoning and external tool integration required for tool-integrated
mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn
direct preference learning framework, tailored for this context, that leverages
feedback from code interpreters and optimizes trajectory-level preferences.
This framework includes multi-turn DPO and multi-turn KTO as specific
implementations. The effectiveness of our framework is validated through
training of various language models using an augmented prompt set from the
GSM8K and MATH datasets. Our results demonstrate substantial improvements: a
supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5%
to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B
model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH.
66db1301c39b38d37f79b11b
Wei Xiong
66db1301c39b38d37f79b11c
Chengshuai Shi
66db1301c39b38d37f79b11d
Jiaming Shen
66db1301c39b38d37f79b11e
Aviv Rosenberg
66db1301c39b38d37f79b11f
Zhen Qin
66db1301c39b38d37f79b120
Daniele Calandriello
66db1301c39b38d37f79b121
Misha Khalman
66db1301c39b38d37f79b122
Rishabh Joshi
66db1301c39b38d37f79b123
Bilal Piot
66db1301c39b38d37f79b124
Mohammad Saleh
66db1301c39b38d37f79b125
Chi Jin
66db1301c39b38d37f79b126
Tong Zhang
66db1301c39b38d37f79b127
Tianqi Liu
extracted_confirmed
2024-09-06T14:45:51.022Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
Building Math Agents with Multi-Turn Iterative Preference Learning
Recent studies have shown that large language models' (LLMs) mathematical
problem-solving capabilities can be enhanced by integrating external tools,
such as code interpreters, and employing multi-turn Chain-of-Thought (CoT)
reasoning. While current methods focus on synthetic data generation and
Supervised Fine-Tuning (SFT), this paper studies the complementary direct
preference learning approach to further improve model performance. However,
existing direct preference learning algorithms are originally designed for the
single-turn chat task, and do not fully address the complexities of multi-turn
reasoning and external tool integration required for tool-integrated
mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn
direct preference learning framework, tailored for this context, that leverages
feedback from code interpreters and optimizes trajectory-level preferences.
This framework includes multi-turn DPO and multi-turn KTO as specific
implementations. The effectiveness of our framework is validated through
training of various language models using an augmented prompt set from the
GSM8K and MATH datasets. Our results demonstrate substantial improvements: a
supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5%
to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B
model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH.
A multi-turn direct preference learning framework, incorporating feedback from code interpreters, enhances large language models' mathematical problem-solving capabilities with substantial performance improvements.
large language models Chain-of-Thought reasoning Supervised Fine-Tuning direct preference learning multi-turn DPO KTO trajectory-level preferences GSM8K MATH Gemma-1.1-it-7B Gemma-2-it-9B