Hierarchical Budget Policy Optimization for Adaptive Reasoning
Large reasoning models achieve remarkable performance through extensive
chain-of-thought generation, yet exhibit significant computational inefficiency
by applying uniform reasoning strategies regardless of problem complexity. We
present Hierarchical Budget Policy Optimization (HBPO), a reinforcement
learning framework that enables models to learn problem-specific reasoning
depths without sacrificing capability. HBPO addresses the fundamental challenge
of exploration space collapse in efficiency-oriented training, where penalties
on long output length systematically bias models away from necessary long
reasoning paths. Through hierarchical budget exploration, our approach
partitions rollout samples into multiple subgroups with distinct token budgets,
aiming to enable efficient resource allocation while preventing degradation of
capability. We introduce differentiated reward mechanisms that create
budget-aware incentives aligned with the complexity of the problem, allowing
models to discover natural correspondences between task requirements and
computational effort. Extensive experiments demonstrate that HBPO reduces
average token usage by up to 60.6% while improving accuracy by 3.14% across
four reasoning benchmarks. Unlike existing methods that impose external
constraints or rely on discrete mode selection, HBPO exhibits emergent adaptive
behavior where models automatically adjust reasoning depth based on problem
complexity. Our results suggest that reasoning efficiency and capability are
not inherently conflicting, and can be simultaneously optimized through
appropriately structured hierarchical training that preserves exploration
diversity.
687ef9a833947f780d9b4ad3
Shangke Lyu
claimed_verified
2025-07-25T08:39:24.100Z
6572a479b3d8dd7b92212a4e
/avatars/fc6d60211504547113a6e14e15ddb4fb.svg
lvshangke
user
paradox122
687ef9a833947f780d9b4ad4
Linjuan Wu
687ef9a833947f780d9b4ad5
Yuchen Yan
claimed_verified
2025-07-22T07:49:03.191Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
687ef9a833947f780d9b4ad6
Xingyu Wu
claimed_verified
2025-07-25T13:36:32.198Z
674d103d2d93658f850151a6
/avatars/e221d43d3d8601d94cb6a44610054e23.svg
AXy1527
user
wuxingyu
687ef9a833947f780d9b4ad7
Hao Li
687ef9a833947f780d9b4ad8
Yongliang Shen
claimed_verified
2025-07-22T07:48:59.928Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
687ef9a833947f780d9b4ad9
Peisheng Jiang
687ef9a833947f780d9b4ada
Weiming Lu
687ef9a833947f780d9b4adb
Jun Xiao
687ef9a833947f780d9b4adc
Yueting Zhuang
Hierarchical Budget Policy Optimization for Adaptive Reasoning
Large reasoning models achieve remarkable performance through extensive
chain-of-thought generation, yet exhibit significant computational inefficiency
by applying uniform reasoning strategies regardless of problem complexity. We
present Hierarchical Budget Policy Optimization (HBPO), a reinforcement
learning framework that enables models to learn problem-specific reasoning
depths without sacrificing capability. HBPO addresses the fundamental challenge
of exploration space collapse in efficiency-oriented training, where penalties
on long output length systematically bias models away from necessary long
reasoning paths. Through hierarchical budget exploration, our approach
partitions rollout samples into multiple subgroups with distinct token budgets,
aiming to enable efficient resource allocation while preventing degradation of
capability. We introduce differentiated reward mechanisms that create
budget-aware incentives aligned with the complexity of the problem, allowing
models to discover natural correspondences between task requirements and
computational effort. Extensive experiments demonstrate that HBPO reduces
average token usage by up to 60.6% while improving accuracy by 3.14% across
four reasoning benchmarks. Unlike existing methods that impose external
constraints or rely on discrete mode selection, HBPO exhibits emergent adaptive
behavior where models automatically adjust reasoning depth based on problem
complexity. Our results suggest that reasoning efficiency and capability are
not inherently conflicting, and can be simultaneously optimized through
appropriately structured hierarchical training that preserves exploration
diversity.
Hierarchical Budget Policy Optimization (HBPO) is a reinforcement learning framework that optimizes reasoning depth for large models, improving efficiency and accuracy by adapting to problem complexity.
Hierarchical Budget Policy Optimization HBPO reinforcement learning chain-of-thought generation computational inefficiency problem-specific reasoning depths exploration space collapse token budgets differentiated reward mechanisms reasoning efficiency capability hierarchical training exploration diversity