Self-Improving LLM Agents at Test-Time
One paradigm of language model (LM) fine-tuning relies on creating large
training datasets, under the assumption that high quantity and diversity will
enable models to generalize to novel tasks after post-training. In practice,
gathering large sets of data is inefficient, and training on them is
prohibitively expensive; worse, there is no guarantee that the resulting model
will handle complex scenarios or generalize better. Moreover, existing
techniques rarely assess whether a training sample provides novel information
or is redundant with the knowledge already acquired by the model, resulting in
unnecessary costs. In this work, we explore a new test-time self-improvement
method to create more effective and generalizable agentic LMs on-the-fly. The
proposed algorithm can be summarized in three steps: (i) first it identifies
the samples that model struggles with (self-awareness), (ii) then generates
similar examples from detected uncertain samples (self-data augmentation), and
(iii) uses these newly generated samples at test-time fine-tuning
(self-improvement). We study two variants of this approach: Test-Time
Self-Improvement (TT-SI), where the same model generates additional training
examples from its own uncertain cases and then learns from them, and contrast
this approach with Test-Time Distillation (TT-D), where a stronger model
generates similar examples for uncertain cases, enabling student to adapt using
distilled supervision. Empirical evaluations across different agent benchmarks
demonstrate that TT-SI improves the performance with +5.48% absolute accuracy
gain on average across all benchmarks and surpasses other standard learning
methods, yet using 68x less training samples. Our findings highlight the
promise of TT-SI, demonstrating the potential of self-improvement algorithms at
test-time as a new paradigm for building more capable agents toward
self-evolution.
68edaaf5de1fee572713a6e4
Emre Can Acikgoz
claimed_verified
2025-10-14T07:31:08.327Z
63888d3fd68e37abd599f428
https://cdn-avatars.huggingface.co/v1/production/uploads/63888d3fd68e37abd599f428/YaNyxG_oM6IgrHTkFZ6Eq.jpeg
emre can
user
emrecanacikgoz
68edaaf5de1fee572713a6e5
Cheng Qian
68edaaf5de1fee572713a6e6
Heng Ji
68edaaf5de1fee572713a6e7
Dilek Hakkani-Tür
68edaaf5de1fee572713a6e8
Gokhan Tur
Self-Improving LLM Agents at Test-Time
One paradigm of language model (LM) fine-tuning relies on creating large
training datasets, under the assumption that high quantity and diversity will
enable models to generalize to novel tasks after post-training. In practice,
gathering large sets of data is inefficient, and training on them is
prohibitively expensive; worse, there is no guarantee that the resulting model
will handle complex scenarios or generalize better. Moreover, existing
techniques rarely assess whether a training sample provides novel information
or is redundant with the knowledge already acquired by the model, resulting in
unnecessary costs. In this work, we explore a new test-time self-improvement
method to create more effective and generalizable agentic LMs on-the-fly. The
proposed algorithm can be summarized in three steps: (i) first it identifies
the samples that model struggles with (self-awareness), (ii) then generates
similar examples from detected uncertain samples (self-data augmentation), and
(iii) uses these newly generated samples at test-time fine-tuning
(self-improvement). We study two variants of this approach: Test-Time
Self-Improvement (TT-SI), where the same model generates additional training
examples from its own uncertain cases and then learns from them, and contrast
this approach with Test-Time Distillation (TT-D), where a stronger model
generates similar examples for uncertain cases, enabling student to adapt using
distilled supervision. Empirical evaluations across different agent benchmarks
demonstrate that TT-SI improves the performance with +5.48% absolute accuracy
gain on average across all benchmarks and surpasses other standard learning
methods, yet using 68x less training samples. Our findings highlight the
promise of TT-SI, demonstrating the potential of self-improvement algorithms at
test-time as a new paradigm for building more capable agents toward
self-evolution.
A test-time self-improvement method enhances language models by generating additional training examples from uncertain cases, leading to better performance with fewer samples.
language model fine-tuning self-awareness self-data augmentation test-time fine-tuning Test-Time Self-Improvement (TT-SI) Test-Time Distillation (TT-D) self-improvement algorithms self-evolution