A Probabilistic Inference Approach to Inference-Time Scaling of LLMs
  using Particle-Based Monte Carlo Methods
Large language models (LLMs) have achieved significant performance gains via
scaling up model sizes and/or data. However, recent evidence suggests
diminishing returns from such approaches, motivating scaling the computation
spent at inference time. Existing inference-time scaling methods, usually with
reward models, cast the task as a search problem, which tends to be vulnerable
to reward hacking as a consequence of approximation errors in reward models. In
this paper, we instead cast inference-time scaling as a probabilistic inference
task and leverage sampling-based techniques to explore the typical set of the
state distribution of a state-space model with an approximate likelihood,
rather than optimize for its mode directly. We propose a novel inference-time
scaling approach by adapting particle-based Monte Carlo methods to this task.
Our empirical evaluation demonstrates that our methods have a 4-16x better
scaling rate over our deterministic search counterparts on various challenging
mathematical reasoning tasks. Using our approach, we show that
Qwen2.5-Math-1.5B-Instruct can surpass GPT-4o accuracy in only 4 rollouts,
while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts.
Our work not only presents an effective method to inference-time scaling, but
also connects the rich literature in probabilistic inference with
inference-time scaling of LLMs to develop more robust algorithms in future
work. Code and further information is available at
https://probabilistic-inference-scaling.github.io.
67a438d26bb8caaab06f5a5e
Isha Puri
claimed_verified
2025-07-29T07:12:22.220Z
62afb707a5bd7cef3e18ed53
/avatars/46092d23509aefdccfef4aacd327b40d.svg
Isha Puri
user
ishapuri
67a438d26bb8caaab06f5a5f
Shivchander Sudalairaj
67a438d26bb8caaab06f5a60
Guangxuan Xu
admin_assigned
2025-02-06T16:07:05.566Z
66104696134c832243bde60d
/avatars/b5a0d194d0e12c60fc5599f81f75c205.svg
Guangxuan Xu
user
gx-ai-architect
67a438d26bb8caaab06f5a61
Kai Xu
67a438d26bb8caaab06f5a62
Akash Srivastava
A Probabilistic Inference Approach to Inference-Time Scaling of LLMs
  using Particle-Based Monte Carlo Methods
Large language models (LLMs) have achieved significant performance gains via
scaling up model sizes and/or data. However, recent evidence suggests
diminishing returns from such approaches, motivating scaling the computation
spent at inference time. Existing inference-time scaling methods, usually with
reward models, cast the task as a search problem, which tends to be vulnerable
to reward hacking as a consequence of approximation errors in reward models. In
this paper, we instead cast inference-time scaling as a probabilistic inference
task and leverage sampling-based techniques to explore the typical set of the
state distribution of a state-space model with an approximate likelihood,
rather than optimize for its mode directly. We propose a novel inference-time
scaling approach by adapting particle-based Monte Carlo methods to this task.
Our empirical evaluation demonstrates that our methods have a 4-16x better
scaling rate over our deterministic search counterparts on various challenging
mathematical reasoning tasks. Using our approach, we show that
Qwen2.5-Math-1.5B-Instruct can surpass GPT-4o accuracy in only 4 rollouts,
while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts.
Our work not only presents an effective method to inference-time scaling, but
also connects the rich literature in probabilistic inference with
inference-time scaling of LLMs to develop more robust algorithms in future
work. Code and further information is available at
https://probabilistic-inference-scaling.github.io.
The paper proposes an inference-time scaling method for LLMs using particle-based Monte Carlo methods, which yields better scaling rates and robustness compared to deterministic search methods.
large language models (LLMs) inference-time scaling probabilistic inference particle-based Monte Carlo methods state-space model approximate likelihood reward models reward hacking rollouts