TreePO: Bridging the Gap of Policy Optimization and Efficacy and
  Inference Efficiency with Heuristic Tree-based Modeling
Recent advancements in aligning large language models via reinforcement
learning have achieved remarkable gains in solving complex reasoning problems,
but at the cost of expensive on-policy rollouts and limited exploration of
diverse reasoning paths. In this work, we introduce TreePO, involving a
self-guided rollout algorithm that views sequence generation as a
tree-structured searching process. Composed of dynamic tree sampling policy and
fixed-length segment decoding, TreePO leverages local uncertainty to warrant
additional branches. By amortizing computation across common prefixes and
pruning low-value paths early, TreePO essentially reduces the per-update
compute burden while preserving or enhancing exploration diversity. Key
contributions include: (1) a segment-wise sampling algorithm that alleviates
the KV cache burden through contiguous segments and spawns new branches along
with an early-stop mechanism; (2) a tree-based segment-level advantage
estimation that considers both global and local proximal policy optimization.
and (3) analysis on the effectiveness of probability and quality-driven dynamic
divergence and fallback strategy. We empirically validate the performance gain
of TreePO on a set reasoning benchmarks and the efficiency saving of GPU hours
from 22\% up to 43\% of the sampling design for the trained models, meanwhile
showing up to 40\% reduction at trajectory-level and 35\% at token-level
sampling compute for the existing models. While offering a free lunch of
inference efficiency, TreePO reveals a practical path toward scaling RL-based
post-training with fewer samples and less compute. Home page locates at
https://m-a-p.ai/TreePO.
68ae7d1a364411bea07df7be
Yizhi Li
claimed_verified
2025-08-27T07:10:35.933Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
68ae7d1a364411bea07df7bf
Qingshui Gu
68ae7d1a364411bea07df7c0
Zhoufutu Wen
claimed_verified
2025-08-27T14:59:44.781Z
6761461f6b3a8119fec25257
/avatars/559321b3dcf8d39b5f6e17a158f33e8e.svg
Maxwell Wen
user
MaxwellWen
68ae7d1a364411bea07df7c1
Ziniu Li
68ae7d1a364411bea07df7c2
Tianshun Xing
claimed_verified
2025-09-03T08:29:49.544Z
65d2251f98b4a470bf6a26e3
https://cdn-avatars.huggingface.co/v1/production/uploads/65d2251f98b4a470bf6a26e3/C4T0LHYGejrI9mu_k3M8p.jpeg
xts
user
xtsssss
68ae7d1a364411bea07df7c3
Shuyue Guo
68ae7d1a364411bea07df7c4
Tianyu Zheng
claimed_verified
2025-08-27T07:10:39.951Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
68ae7d1a364411bea07df7c5
Xin Zhou
68ae7d1a364411bea07df7c6
Xingwei Qu
68ae7d1a364411bea07df7c7
Wangchunshu Zhou
admin_assigned
2025-08-27T15:09:37.920Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
68ae7d1a364411bea07df7c8
Zheng Zhang
68ae7d1a364411bea07df7c9
Wei Shen
68ae7d1a364411bea07df7ca
Qian Liu
68ae7d1a364411bea07df7cb
Chenghua Lin
admin_assigned
2025-08-27T15:09:54.726Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
68ae7d1a364411bea07df7cc
Jian Yang
68ae7d1a364411bea07df7cd
Ge Zhang
claimed_verified
2025-08-27T07:10:37.920Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
68ae7d1a364411bea07df7ce
Wenhao Huang
TreePO: Bridging the Gap of Policy Optimization and Efficacy and
  Inference Efficiency with Heuristic Tree-based Modeling
Recent advancements in aligning large language models via reinforcement
learning have achieved remarkable gains in solving complex reasoning problems,
but at the cost of expensive on-policy rollouts and limited exploration of
diverse reasoning paths. In this work, we introduce TreePO, involving a
self-guided rollout algorithm that views sequence generation as a
tree-structured searching process. Composed of dynamic tree sampling policy and
fixed-length segment decoding, TreePO leverages local uncertainty to warrant
additional branches. By amortizing computation across common prefixes and
pruning low-value paths early, TreePO essentially reduces the per-update
compute burden while preserving or enhancing exploration diversity. Key
contributions include: (1) a segment-wise sampling algorithm that alleviates
the KV cache burden through contiguous segments and spawns new branches along
with an early-stop mechanism; (2) a tree-based segment-level advantage
estimation that considers both global and local proximal policy optimization.
and (3) analysis on the effectiveness of probability and quality-driven dynamic
divergence and fallback strategy. We empirically validate the performance gain
of TreePO on a set reasoning benchmarks and the efficiency saving of GPU hours
from 22\% up to 43\% of the sampling design for the trained models, meanwhile
showing up to 40\% reduction at trajectory-level and 35\% at token-level
sampling compute for the existing models. While offering a free lunch of
inference efficiency, TreePO reveals a practical path toward scaling RL-based
post-training with fewer samples and less compute. Home page locates at
https://m-a-p.ai/TreePO.
TreePO, a self-guided rollout algorithm for sequence generation, reduces computational cost and enhances exploration diversity in reinforcement learning for large language models.
reinforcement learning sequence generation tree-structured searching dynamic tree sampling policy fixed-length segment decoding local uncertainty computation amortization low-value path pruning segment-wise sampling KV cache burden tree-based segment-level advantage estimation proximal policy optimization probability-driven dynamic divergence quality-driven fallback strategy trajectory-level sampling token-level sampling