Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with
  LLMs
Large Language Models (LLMs) employ multi-turn interaction as a fundamental
paradigm for completing complex tasks. However, their performance often
degrades in extended interactions, as they are typically trained on static,
single-turn data, which hinders their ability to adapt to real-time user
feedback. To address this limitation, we first propose a new paradigm:
Test-Time Policy Adaptation for Multi-Turn Interactions (T2PAM), which utilizes
user feedback from the ongoing interaction as a reward signal to estimate a
latent optimal policy aligned with user preferences, then updates a small
subset of parameters to steer the model toward this policy, ultimately enabling
efficient in-conversation self-correction. We then introduce Optimum-Referenced
One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes T2PAM.
ROSA guides the model parameters toward a theoretical optimal policy in a
single, efficient update step, avoiding costly iterative gradient-based
optimization and minimizing computational overhead. We provide a rigorous
theoretical analysis guaranteeing that the policy of ROSA converges to the
preference of user as the number of interactions increases. Extensive
experiments on challenging benchmark demonstrate that ROSA achieves significant
improvements in both task effectiveness and efficiency.
68dca79d4159d1f2418f9ba8
Chenxing Wei
68dca79d4159d1f2418f9ba9
Hong Wang
68dca79d4159d1f2418f9baa
Ying He
68dca79d4159d1f2418f9bab
Fei Yu
68dca79d4159d1f2418f9bac
Yao Shu
Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with
  LLMs
Large Language Models (LLMs) employ multi-turn interaction as a fundamental
paradigm for completing complex tasks. However, their performance often
degrades in extended interactions, as they are typically trained on static,
single-turn data, which hinders their ability to adapt to real-time user
feedback. To address this limitation, we first propose a new paradigm:
Test-Time Policy Adaptation for Multi-Turn Interactions (T2PAM), which utilizes
user feedback from the ongoing interaction as a reward signal to estimate a
latent optimal policy aligned with user preferences, then updates a small
subset of parameters to steer the model toward this policy, ultimately enabling
efficient in-conversation self-correction. We then introduce Optimum-Referenced
One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes T2PAM.
ROSA guides the model parameters toward a theoretical optimal policy in a
single, efficient update step, avoiding costly iterative gradient-based
optimization and minimizing computational overhead. We provide a rigorous
theoretical analysis guaranteeing that the policy of ROSA converges to the
preference of user as the number of interactions increases. Extensive
experiments on challenging benchmark demonstrate that ROSA achieves significant
improvements in both task effectiveness and efficiency.
ROSA, a lightweight algorithm, enhances multi-turn interactions in LLMs by adapting to user feedback in real-time, improving both task effectiveness and efficiency.
multi-turn interaction Test-Time Policy Adaptation T2PAM latent optimal policy parameter adaptation Optimum-Referenced One-Step Adaptation ROSA theoretical optimal policy gradient-based optimization computational overhead