Language Models are Hidden Reasoners: Unlocking Latent Reasoning
  Capabilities via Self-Rewarding
Large language models (LLMs) have shown impressive capabilities, but still
struggle with complex reasoning tasks requiring multiple steps. While
prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at
inference time, optimizing reasoning capabilities during training remains
challenging. We introduce LaTent Reasoning Optimization (LaTRO), a principled
framework that formulates reasoning as sampling from a latent distribution and
optimizes it via variational approaches. LaTRO enables LLMs to concurrently
improve both their reasoning process and ability to evaluate reasoning quality,
without requiring external feedback or reward models. We validate LaTRO through
experiments on GSM8K and ARC-Challenge datasets using multiple model
architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of
12.5% over base models and 9.6% over supervised fine-tuning across
Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that
pre-trained LLMs possess latent reasoning capabilities that can be unlocked and
enhanced through our proposed optimization approach in a self-improvement
manner. The code of LaTRO is available at
https://github.com/SalesforceAIResearch/LaTRO.
6732834b2c3d3a050a86288a
Haolin Chen
extracted_pending
2024-11-11T22:21:00.560Z
65e0c21ebd0c59bbf271004c
/avatars/3210f41fd433d798fd7857c943288625.svg
Haolin Chen
user
hlnchen
6732834b2c3d3a050a86288b
Yihao Feng
6732834b2c3d3a050a86288c
Zuxin Liu
claimed_verified
2024-11-15T09:27:25.665Z
638e971f2cc490759fe9210e
/avatars/9578d0badc084d340c028a59e571abd3.svg
Liu
user
Zuxin
6732834b2c3d3a050a86288d
Weiran Yao
claimed_verified
2024-11-15T09:27:23.471Z
632cdea254e2c512c8f95b12
/avatars/a6d06cdd75861ae7d589f1343d81a5c5.svg
Weiran Yao
user
weirayao
6732834b2c3d3a050a86288e
Akshara Prabhakar
claimed_verified
2025-04-04T07:11:17.227Z
6475718de0b188d3cb25f346
/avatars/b0cdca7e19bbc1dbac00ea664108b51e.svg
Akshara Prabhakar
user
aksh555
6732834b2c3d3a050a86288f
Shelby Heinecke
6732834b2c3d3a050a862890
Ricky Ho
6732834b2c3d3a050a862891
Phil Mui
6732834b2c3d3a050a862892
Silvio Savarese
6732834b2c3d3a050a862893
Caiming Xiong
6732834b2c3d3a050a862894
Huan Wang
Language Models are Hidden Reasoners: Unlocking Latent Reasoning
  Capabilities via Self-Rewarding
Large language models (LLMs) have shown impressive capabilities, but still
struggle with complex reasoning tasks requiring multiple steps. While
prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at
inference time, optimizing reasoning capabilities during training remains
challenging. We introduce LaTent Reasoning Optimization (LaTRO), a principled
framework that formulates reasoning as sampling from a latent distribution and
optimizes it via variational approaches. LaTRO enables LLMs to concurrently
improve both their reasoning process and ability to evaluate reasoning quality,
without requiring external feedback or reward models. We validate LaTRO through
experiments on GSM8K and ARC-Challenge datasets using multiple model
architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of
12.5% over base models and 9.6% over supervised fine-tuning across
Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that
pre-trained LLMs possess latent reasoning capabilities that can be unlocked and
enhanced through our proposed optimization approach in a self-improvement
manner. The code of LaTRO is available at
https://github.com/SalesforceAIResearch/LaTRO.
LaTent Reasoning Optimization (LaTRO) enhances LLMs' reasoning capabilities through variational optimization of latent distributions, improving zero-shot accuracy on complex reasoning tasks.
Large language models (LLMs) Chain-of-Thought (CoT) LaTent Reasoning Optimization (LaTRO) latent distribution variational approaches GSM8K ARC-Challenge zero-shot accuracy Phi-3.5-mini Mistral-7B Llama-3.1-8B pre-trained LLMs latent reasoning capabilities self-improvement