DiaTool-DPO: Multi-Turn Direct Preference Optimization for
  Tool-Augmented Large Language Models
Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in
real-world applications, but face challenges in handling incomplete queries and
out-of-scope requests. While existing approaches rely mainly on Supervised
Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method
that enhances TA-LLM's dialogue capabilities through Direct Preference
Optimization. We model TA-LLM interactions as a Markov Decision Process with 5
distinct dialogue states and categorize user queries into 3 types based on
their state transition trajectories. We automatically construct paired
trajectory datasets of correct and incorrect dialogue flows and introduce a
specialized objective loss for dialogue control. Our comprehensive evaluation
demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in
information gathering, 91% in tool call rejection) with substantial
improvements over baseline (44% and 9.6% respectively) while maintaining core
functionality. Our approach opens new possibilities for developing TA-LLMs that
can handle diverse real-world scenarios without requiring additional expert
demonstrations or human labeling.
67f4d15a8f00d281c155880f
Sunghee Jung
claimed_verified
2025-04-08T10:41:40.756Z
65e30342e8b017ee1384824c
/avatars/e5d07b037f611ccfaf719959d971d102.svg
Sunghee Jung
user
hash2430
67f4d15a8f00d281c1558810
Donghun Lee
67f4d15a8f00d281c1558811
Shinbok Lee
67f4d15a8f00d281c1558812
Gaeun Seo
admin_assigned
2025-04-08T10:42:12.004Z
63148a8f5f47a18962765802
/avatars/bc58a863727794006dddf758efa09411.svg
gaeunseo
user
gaeunseo
67f4d15a8f00d281c1558813
Daniel Lee
67f4d15a8f00d281c1558814
Byeongil Ko
67f4d15a8f00d281c1558815
Junrae Cho
admin_assigned
2025-04-08T10:42:51.801Z
6464d7e4f4f3102bb6c79268
/avatars/177e0c0f49728f8554173f63cc52a859.svg
cho
user
junrae
67f4d15a8f00d281c1558816
Kihyun Kim
67f4d15a8f00d281c1558817
Eunggyun Kim
67f4d15a8f00d281c1558818
Myeongcheol Shin
admin_assigned
2025-04-08T10:43:26.136Z
60c89e7fd9d94022e7eac9c4
/avatars/0a5dedfffcfb9c9ca7034c3e734c120d.svg
Myeongcheol Shin
user
dsindex
DiaTool-DPO: Multi-Turn Direct Preference Optimization for
  Tool-Augmented Large Language Models
Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in
real-world applications, but face challenges in handling incomplete queries and
out-of-scope requests. While existing approaches rely mainly on Supervised
Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method
that enhances TA-LLM's dialogue capabilities through Direct Preference
Optimization. We model TA-LLM interactions as a Markov Decision Process with 5
distinct dialogue states and categorize user queries into 3 types based on
their state transition trajectories. We automatically construct paired
trajectory datasets of correct and incorrect dialogue flows and introduce a
specialized objective loss for dialogue control. Our comprehensive evaluation
demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in
information gathering, 91% in tool call rejection) with substantial
improvements over baseline (44% and 9.6% respectively) while maintaining core
functionality. Our approach opens new possibilities for developing TA-LLMs that
can handle diverse real-world scenarios without requiring additional expert
demonstrations or human labeling.
DiaTool-DPO improves TA-LLM dialogue capabilities through Direct Preference Optimization, enhancing performance in information gathering and tool call rejection compared to baselines.
Supervised Fine-Tuning Direct Preference Optimization Markov Decision Process dialogue states user queries dialogue control information gathering tool call rejection TA-LLMs