QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
Language agents have become a promising solution to complex interactive
tasks. One of the key ingredients to the success of language agents is the
reward model on the trajectory of the agentic workflow, which provides valuable
guidance during training or inference. However, due to the lack of annotations
of intermediate interactions, most existing works use an outcome reward model
to optimize policies across entire trajectories. This may lead to sub-optimal
policies and hinder the overall performance. To address this, we propose QLASS
(Q-guided Language Agent Stepwise Search), to automatically generate
annotations by estimating Q-values in a stepwise manner for open language
agents. By introducing a reasoning tree and performing process reward modeling,
QLASS provides effective intermediate guidance for each step. With the stepwise
guidance, we propose a Q-guided generation strategy to enable language agents
to better adapt to long-term value, resulting in significant performance
improvement during model inference on complex interactive agent tasks. Notably,
even with almost half the annotated data, QLASS retains strong performance,
demonstrating its efficiency in handling limited supervision. We also
empirically demonstrate that QLASS can lead to more effective decision making
through qualitative analysis. We will release our code and data.
67a2d59fd5ad3369a66ff394
Zongyu Lin
67a2d59fd5ad3369a66ff395
Yao Tang
claimed_verified
2025-06-10T09:31:20.416Z
667119d6578448466d9531a6
/avatars/72c31909a5584b1306b6404b94a22b2a.svg
Yao Tang
user
YaoTang23
67a2d59fd5ad3369a66ff396
Xingcheng Yao
67a2d59fd5ad3369a66ff397
Da Yin
67a2d59fd5ad3369a66ff398
Ziniu Hu
67a2d59fd5ad3369a66ff399
Yizhou Sun
67a2d59fd5ad3369a66ff39a
Kai-Wei Chang
QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
Language agents have become a promising solution to complex interactive
tasks. One of the key ingredients to the success of language agents is the
reward model on the trajectory of the agentic workflow, which provides valuable
guidance during training or inference. However, due to the lack of annotations
of intermediate interactions, most existing works use an outcome reward model
to optimize policies across entire trajectories. This may lead to sub-optimal
policies and hinder the overall performance. To address this, we propose QLASS
(Q-guided Language Agent Stepwise Search), to automatically generate
annotations by estimating Q-values in a stepwise manner for open language
agents. By introducing a reasoning tree and performing process reward modeling,
QLASS provides effective intermediate guidance for each step. With the stepwise
guidance, we propose a Q-guided generation strategy to enable language agents
to better adapt to long-term value, resulting in significant performance
improvement during model inference on complex interactive agent tasks. Notably,
even with almost half the annotated data, QLASS retains strong performance,
demonstrating its efficiency in handling limited supervision. We also
empirically demonstrate that QLASS can lead to more effective decision making
through qualitative analysis. We will release our code and data.
QLASS uses Q-value estimation for stepwise guidance in language agents, improving performance with limited annotated data.
reward model Q-values reasoning tree process reward modeling Q-guided generation strategy long-term value