Thinker: Learning to Think Fast and Slow
Recent studies show that the reasoning capabilities of Large Language Models
(LLMs) can be improved by applying Reinforcement Learning (RL) to
question-answering (QA) tasks in areas such as math and coding. With a long
context length, LLMs may learn to perform search, as indicated by the
self-correction behavior observed in DeepSeek R1. However, this search behavior
is often imprecise and lacks confidence, resulting in long, redundant responses
and highlighting deficiencies in intuition and verification. Inspired by the
Dual Process Theory in psychology, we introduce a simple modification to the QA
task that includes four stages: Fast Thinking, where the LLM must answer within
a strict token budget; Verification, where the model evaluates its initial
response; Slow Thinking, where it refines the initial response with more
deliberation; and Summarization, where it distills the refinement from the
previous stage into precise steps. Our proposed task improves average accuracy
from 24.9% to 27.9% for Qwen2.5-1.5B, and from 45.9% to 49.8% for
DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone
achieves 26.8% accuracy using fewer than 1000 tokens, demonstrating substantial
inference efficiency gains. These findings suggest that intuition and
deliberative reasoning are distinct, complementary systems benefiting from
targeted training.
6837361b4b3a674ac2a2c27a
Stephen Chung
6837361b4b3a674ac2a2c27b
Wenyu Du
6837361b4b3a674ac2a2c27c
Jie Fu
claimed_verified
2025-06-04T09:03:46.383Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
Thinker: Learning to Think Fast and Slow
Recent studies show that the reasoning capabilities of Large Language Models
(LLMs) can be improved by applying Reinforcement Learning (RL) to
question-answering (QA) tasks in areas such as math and coding. With a long
context length, LLMs may learn to perform search, as indicated by the
self-correction behavior observed in DeepSeek R1. However, this search behavior
is often imprecise and lacks confidence, resulting in long, redundant responses
and highlighting deficiencies in intuition and verification. Inspired by the
Dual Process Theory in psychology, we introduce a simple modification to the QA
task that includes four stages: Fast Thinking, where the LLM must answer within
a strict token budget; Verification, where the model evaluates its initial
response; Slow Thinking, where it refines the initial response with more
deliberation; and Summarization, where it distills the refinement from the
previous stage into precise steps. Our proposed task improves average accuracy
from 24.9% to 27.9% for Qwen2.5-1.5B, and from 45.9% to 49.8% for
DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone
achieves 26.8% accuracy using fewer than 1000 tokens, demonstrating substantial
inference efficiency gains. These findings suggest that intuition and
deliberative reasoning are distinct, complementary systems benefiting from
targeted training.
A four-stage QA task modification, inspired by Dual Process Theory, improves the accuracy of LLMs in math and coding by separating intuition and deliberation.
Reinforcement Learning (RL) Question-Answering (QA) DeepSeek R1 Dual Process Theory Fast Thinking Verification Slow Thinking Summarization