Goal Alignment in LLM-Based User Simulators for Conversational AI
User simulators are essential to conversational AI, enabling scalable agent
development and evaluation through simulated interactions. While current Large
Language Models (LLMs) have advanced user simulation capabilities, we reveal
that they struggle to consistently demonstrate goal-oriented behavior across
multi-turn conversations--a critical limitation that compromises their
reliability in downstream applications. We introduce User Goal State Tracking
(UGST), a novel framework that tracks user goal progression throughout
conversations. Leveraging UGST, we present a three-stage methodology for
developing user simulators that can autonomously track goal progression and
reason to generate goal-aligned responses. Moreover, we establish comprehensive
evaluation metrics for measuring goal alignment in user simulators, and
demonstrate that our approach yields substantial improvements across two
benchmarks (MultiWOZ 2.4 and {\tau}-Bench). Our contributions address a
critical gap in conversational AI and establish UGST as an essential framework
for developing goal-aligned user simulators.
688902c566e37b2a6b292005
Shuhaib Mehri
688902c566e37b2a6b292006
Xiaocheng Yang
688902c566e37b2a6b292007
Takyoung Kim
688902c566e37b2a6b292008
Gokhan Tur
688902c566e37b2a6b292009
Shikib Mehri
688902c566e37b2a6b29200a
Dilek Hakkani-Tür
Goal Alignment in LLM-Based User Simulators for Conversational AI
User simulators are essential to conversational AI, enabling scalable agent
development and evaluation through simulated interactions. While current Large
Language Models (LLMs) have advanced user simulation capabilities, we reveal
that they struggle to consistently demonstrate goal-oriented behavior across
multi-turn conversations--a critical limitation that compromises their
reliability in downstream applications. We introduce User Goal State Tracking
(UGST), a novel framework that tracks user goal progression throughout
conversations. Leveraging UGST, we present a three-stage methodology for
developing user simulators that can autonomously track goal progression and
reason to generate goal-aligned responses. Moreover, we establish comprehensive
evaluation metrics for measuring goal alignment in user simulators, and
demonstrate that our approach yields substantial improvements across two
benchmarks (MultiWOZ 2.4 and {\tau}-Bench). Our contributions address a
critical gap in conversational AI and establish UGST as an essential framework
for developing goal-aligned user simulators.
A novel framework, User Goal State Tracking (UGST), is introduced to improve goal-aligned behavior in user simulators for conversational AI, demonstrating significant improvements in goal alignment across benchmarks.
User Goal State Tracking UGST goal-oriented behavior multi-turn conversations user simulators conversational AI goal-aligned responses evaluation metrics MultiWOZ 2.4 τ-Bench