TTRV: Test-Time Reinforcement Learning for Vision Language Models
Existing methods for extracting reward signals in Reinforcement Learning
typically rely on labeled data and dedicated training splits, a setup that
contrasts with how humans learn directly from their environment. In this work,
we propose TTRV to enhance vision language understanding by adapting the model
on the fly at inference time, without the need for any labeled data.
Concretely, we enhance the Group Relative Policy Optimization (GRPO) framework
by designing rewards based on the frequency of the base model's output, while
inferring on each test sample multiple times. Further, we also propose to
control the diversity of the model's output by simultaneously rewarding the
model for obtaining low entropy of the output empirical distribution. Our
approach delivers consistent gains across both object recognition and visual
question answering (VQA), with improvements of up to 52.4% and 29.8%,
respectively, and average boosts of 24.6% and 10.0% across 16
datasets.Remarkably, on image recognition, TTRV applied to InternVL 8B
surpasses GPT-4o by an average of 2.3% over 8 benchmarks, while remaining
highly competitive on VQA, demonstrating that test-time reinforcement learning
can match or exceed the strongest proprietary models. Finally, we find many
interesting properties of test-time RL for VLMs: for example, even in extremely
data-constrained scenarios, where adaptation is performed on a single randomly
chosen unlabeled test example, TTRV still yields non-trivial improvements of up
to 5.5% in recognition tasks.
68e71bfa7ae125f9582e6991
Akshit Singh
68e71bfa7ae125f9582e6992
Shyam Marjit
68e71bfa7ae125f9582e6993
Wei Lin
68e71bfa7ae125f9582e6994
Paul Gavrikov
68e71bfa7ae125f9582e6995
Serena Yeung-Levy
68e71bfa7ae125f9582e6996
Hilde Kuehne
68e71bfa7ae125f9582e6997
Rogerio Feris
68e71bfa7ae125f9582e6998
Sivan Doveh
68e71bfa7ae125f9582e6999
James Glass
68e71bfa7ae125f9582e699a
M. Jehanzeb Mirza
TTRV: Test-Time Reinforcement Learning for Vision Language Models
Existing methods for extracting reward signals in Reinforcement Learning
typically rely on labeled data and dedicated training splits, a setup that
contrasts with how humans learn directly from their environment. In this work,
we propose TTRV to enhance vision language understanding by adapting the model
on the fly at inference time, without the need for any labeled data.
Concretely, we enhance the Group Relative Policy Optimization (GRPO) framework
by designing rewards based on the frequency of the base model's output, while
inferring on each test sample multiple times. Further, we also propose to
control the diversity of the model's output by simultaneously rewarding the
model for obtaining low entropy of the output empirical distribution. Our
approach delivers consistent gains across both object recognition and visual
question answering (VQA), with improvements of up to 52.4% and 29.8%,
respectively, and average boosts of 24.6% and 10.0% across 16
datasets.Remarkably, on image recognition, TTRV applied to InternVL 8B
surpasses GPT-4o by an average of 2.3% over 8 benchmarks, while remaining
highly competitive on VQA, demonstrating that test-time reinforcement learning
can match or exceed the strongest proprietary models. Finally, we find many
interesting properties of test-time RL for VLMs: for example, even in extremely
data-constrained scenarios, where adaptation is performed on a single randomly
chosen unlabeled test example, TTRV still yields non-trivial improvements of up
to 5.5% in recognition tasks.
TTRV enhances vision language understanding through test-time reinforcement learning, improving performance on object recognition and VQA without labeled data.
Reinforcement Learning Group Relative Policy Optimization (GRPO) entropy test-time reinforcement learning vision language models (VLMs) InternVL 8B GPT-4o