Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Training models to effectively use test-time compute is crucial for improving
the reasoning performance of LLMs. Current methods mostly do so via fine-tuning
on search traces or running RL with 0/1 outcome reward, but do these approaches
efficiently utilize test-time compute? Would these approaches continue to scale
as the budget improves? In this paper, we try to answer these questions. We
formalize the problem of optimizing test-time compute as a meta-reinforcement
learning (RL) problem, which provides a principled perspective on spending
test-time compute. This perspective enables us to view the long output stream
from the LLM as consisting of several episodes run at test time and leads us to
use a notion of cumulative regret over output tokens as a way to measure the
efficacy of test-time compute. Akin to how RL algorithms can best tradeoff
exploration and exploitation over training, minimizing cumulative regret would
also provide the best balance between exploration and exploitation in the token
stream. While we show that state-of-the-art models do not minimize regret, one
can do so by maximizing a dense reward bonus in conjunction with the outcome
0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in
the output stream, quantified by the change in the likelihood of eventual
success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or
MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT
leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token
efficiency for math reasoning compared to outcome-reward RL.
67d0e38171b6b577dbb8c72c
Yuxiao Qu
claimed_verified
2025-03-12T08:41:33.926Z
6500bbf5e102da55f9ed43fc
https://cdn-avatars.huggingface.co/v1/production/uploads/6500bbf5e102da55f9ed43fc/QZ6EAFV2CStFsILmTJw5D.jpeg
Yuxiao Qu
user
CohenQu
67d0e38171b6b577dbb8c72d
Matthew Y. R. Yang
67d0e38171b6b577dbb8c72e
Amrith Setlur
admin_assigned
2025-03-12T15:54:33.962Z
652b3cf8802e3d1a4fb4669d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/dTL6uoL3Ha_V1cgOrRG5C.png
Amrith Setlur
user
ars22
67d0e38171b6b577dbb8c72f
Lewis Tunstall
admin_assigned
2025-03-12T15:53:47.844Z
5f0c746619cb630495b814fd
https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
Lewis Tunstall
user
lewtun
67d0e38171b6b577dbb8c730
Edward Emanuel Beeching
admin_assigned
2025-03-12T15:54:21.375Z
6200d0a443eb0913fa2df7cc
https://cdn-avatars.huggingface.co/v1/production/uploads/1644220542819-noauth.jpeg
Edward Beeching
user
edbeeching
67d0e38171b6b577dbb8c731
Ruslan Salakhutdinov
67d0e38171b6b577dbb8c732
Aviral Kumar
admin_assigned
2025-03-12T15:54:05.568Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Training models to effectively use test-time compute is crucial for improving
the reasoning performance of LLMs. Current methods mostly do so via fine-tuning
on search traces or running RL with 0/1 outcome reward, but do these approaches
efficiently utilize test-time compute? Would these approaches continue to scale
as the budget improves? In this paper, we try to answer these questions. We
formalize the problem of optimizing test-time compute as a meta-reinforcement
learning (RL) problem, which provides a principled perspective on spending
test-time compute. This perspective enables us to view the long output stream
from the LLM as consisting of several episodes run at test time and leads us to
use a notion of cumulative regret over output tokens as a way to measure the
efficacy of test-time compute. Akin to how RL algorithms can best tradeoff
exploration and exploitation over training, minimizing cumulative regret would
also provide the best balance between exploration and exploitation in the token
stream. While we show that state-of-the-art models do not minimize regret, one
can do so by maximizing a dense reward bonus in conjunction with the outcome
0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in
the output stream, quantified by the change in the likelihood of eventual
success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or
MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT
leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token
efficiency for math reasoning compared to outcome-reward RL.
The paper formalizes test-time compute optimization as a meta-reinforcement learning problem, introducing Meta Reinforcement Fine-Tuning (MRT) to enhance performance and token efficiency in large language models.
meta-reinforcement learning RL test-time compute cumulative regret dense reward bonus likelihood of eventual success Meta Reinforcement Fine-Tuning MRT