Scaling LLM Test-Time Compute Optimally can be More Effective than
  Scaling Model Parameters
Enabling LLMs to improve their outputs by using more test-time computation is
a critical step towards building generally self-improving agents that can
operate on open-ended natural language. In this paper, we study the scaling of
inference-time computation in LLMs, with a focus on answering the question: if
an LLM is allowed to use a fixed but non-trivial amount of inference-time
compute, how much can it improve its performance on a challenging prompt?
Answering this question has implications not only on the achievable performance
of LLMs, but also on the future of LLM pretraining and how one should tradeoff
inference-time and pre-training compute. Despite its importance, little
research attempted to understand the scaling behaviors of various test-time
inference methods. Moreover, current work largely provides negative results for
a number of these strategies. In this work, we analyze two primary mechanisms
to scale test-time computation: (1) searching against dense, process-based
verifier reward models; and (2) updating the model's distribution over a
response adaptively, given the prompt at test time. We find that in both cases,
the effectiveness of different approaches to scaling test-time compute
critically varies depending on the difficulty of the prompt. This observation
motivates applying a "compute-optimal" scaling strategy, which acts to most
effectively allocate test-time compute adaptively per prompt. Using this
compute-optimal strategy, we can improve the efficiency of test-time compute
scaling by more than 4x compared to a best-of-N baseline. Additionally, in a
FLOPs-matched evaluation, we find that on problems where a smaller base model
attains somewhat non-trivial success rates, test-time compute can be used to
outperform a 14x larger model.
66b2d3a09f3dfa895cd036be
Charlie Snell
extracted_confirmed
2024-08-07T01:56:25.477Z
605d07fa8a3450814bada877
/avatars/eafe986982057fbaba962b99d5543477.svg
Charlie Snell
user
sea-snell
66b2d3a09f3dfa895cd036bf
Jaehoon Lee
66b2d3a09f3dfa895cd036c0
Kelvin Xu
66b2d3a09f3dfa895cd036c1
Aviral Kumar
admin_assigned
2024-08-08T12:21:23.161Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
Scaling LLM Test-Time Compute Optimally can be More Effective than
  Scaling Model Parameters
Enabling LLMs to improve their outputs by using more test-time computation is
a critical step towards building generally self-improving agents that can
operate on open-ended natural language. In this paper, we study the scaling of
inference-time computation in LLMs, with a focus on answering the question: if
an LLM is allowed to use a fixed but non-trivial amount of inference-time
compute, how much can it improve its performance on a challenging prompt?
Answering this question has implications not only on the achievable performance
of LLMs, but also on the future of LLM pretraining and how one should tradeoff
inference-time and pre-training compute. Despite its importance, little
research attempted to understand the scaling behaviors of various test-time
inference methods. Moreover, current work largely provides negative results for
a number of these strategies. In this work, we analyze two primary mechanisms
to scale test-time computation: (1) searching against dense, process-based
verifier reward models; and (2) updating the model's distribution over a
response adaptively, given the prompt at test time. We find that in both cases,
the effectiveness of different approaches to scaling test-time compute
critically varies depending on the difficulty of the prompt. This observation
motivates applying a "compute-optimal" scaling strategy, which acts to most
effectively allocate test-time compute adaptively per prompt. Using this
compute-optimal strategy, we can improve the efficiency of test-time compute
scaling by more than 4x compared to a best-of-N baseline. Additionally, in a
FLOPs-matched evaluation, we find that on problems where a smaller base model
attains somewhat non-trivial success rates, test-time compute can be used to
outperform a 14x larger model.
Research on scaling test-time computation in LLMs finds that compute-optimal strategies can enhance performance and efficiency beyond larger base models.
LLMs inference-time computation verifier reward models adaptive response distribution difficulty of prompts compute-optimal scaling strategy FLOPs