O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning
Recently, long-thought reasoning LLMs, such as OpenAI's O1, adopt extended
reasoning processes similar to how humans ponder over complex problems. This
reasoning paradigm significantly enhances the model's problem-solving abilities
and has achieved promising results. However, long-thought reasoning process
leads to a substantial increase in inference time. A pressing challenge is
reducing the inference overhead of long-thought LLMs while ensuring accuracy.
In this paper, we experimentally demonstrate that long-thought reasoning models
struggle to effectively allocate token budgets based on problem difficulty and
reasoning redundancies. To address this, we propose Length-Harmonizing
Fine-Tuning (O1-Pruner), aiming at minimizing reasoning overhead while
maintaining accuracy. This effective fine-tuning method first estimates the
LLM's baseline performance through pre-sampling and then uses RL-style
fine-tuning to encourage the model to generate shorter reasoning processes
under accuracy constraints. This allows the model to achieve efficient
reasoning with lower redundancy while maintaining accuracy. Experiments on
various mathematical reasoning benchmarks show that O1-Pruner not only
significantly reduces inference overhead but also achieves higher accuracy,
providing a novel and promising solution to this challenge. Our code is coming
soon at https://github.com/StarDewXXX/O1-Pruner
6791b165c6fee5f0588773a9
Haotian Luo
claimed_verified
2025-05-02T06:38:58.014Z
632ab8f5a968c34257da5c52
/avatars/59df09e6c9e1e633170514d950ad7981.svg
Haotian Luo
user
LordNoah
6791b165c6fee5f0588773aa
Li Shen
claimed_verified
2025-01-26T11:38:40.905Z
62de356ad89af8c07209e7d4
/avatars/610629958726b270418368b8b7f61469.svg
Li Shen
user
mathshenli
6791b165c6fee5f0588773ab
Haiying He
6791b165c6fee5f0588773ac
Yibo Wang
6791b165c6fee5f0588773ad
Shiwei Liu
claimed_verified
2025-03-03T11:15:27.241Z
65b04d2291e63920a7898c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
Liu
user
Shiweiliuiiiiiii
6791b165c6fee5f0588773ae
Wei Li
6791b165c6fee5f0588773af
Naiqiang Tan
6791b165c6fee5f0588773b0
Xiaochun Cao
admin_assigned
2025-01-23T09:50:53.299Z
64ddc5ea83645d0f98599d6e
/avatars/99a148c29cfbff77ed6150cc279a47c0.svg
caoxiaochun
user
cxc361461518
6791b165c6fee5f0588773b1
Dacheng Tao
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning
Recently, long-thought reasoning LLMs, such as OpenAI's O1, adopt extended
reasoning processes similar to how humans ponder over complex problems. This
reasoning paradigm significantly enhances the model's problem-solving abilities
and has achieved promising results. However, long-thought reasoning process
leads to a substantial increase in inference time. A pressing challenge is
reducing the inference overhead of long-thought LLMs while ensuring accuracy.
In this paper, we experimentally demonstrate that long-thought reasoning models
struggle to effectively allocate token budgets based on problem difficulty and
reasoning redundancies. To address this, we propose Length-Harmonizing
Fine-Tuning (O1-Pruner), aiming at minimizing reasoning overhead while
maintaining accuracy. This effective fine-tuning method first estimates the
LLM's baseline performance through pre-sampling and then uses RL-style
fine-tuning to encourage the model to generate shorter reasoning processes
under accuracy constraints. This allows the model to achieve efficient
reasoning with lower redundancy while maintaining accuracy. Experiments on
various mathematical reasoning benchmarks show that O1-Pruner not only
significantly reduces inference overhead but also achieves higher accuracy,
providing a novel and promising solution to this challenge. Our code is coming
soon at https://github.com/StarDewXXX/O1-Pruner
O1-Pruner, an RL-style fine-tuning method, reduces inference time and enhances accuracy in long-thought reasoning LLMs by optimizing token usage.
LLMs long-thought reasoning token budgets RL-style fine-tuning Length-Harmonizing Fine-Tuning O1-Pruner reasoning redundancies inference overhead