ExPO: Bootstrapping LLM Reasoning without Expert Chain-of-Thought
Ruiyang Zhou, Shuozhe Li, Amy Zhang, Liu Leqi
Recent advances in large language models have been driven by reinforcement learning-style post-training, which improves reasoning by optimizing model outputs based on reward or preference signals.  These methods require positive training samples, which are difficult to obtain: expert demonstrations are costly, and self-generated positive responses are rare during early-stage training, particularly for challenging tasks.In this work, we identify two key properties of effective positive samples: they should have high probability under the current policy and increase the model’s likelihood of predicting the correct answer.Building on these insights, we propose ExPO (Self-Explanation Policy Optimization), a simple and modular framework that generates such positive samples from the model itself. ExPO improves both learning efficiency and final performance on reasoning benchmarks, even outperforming  methods relying on expert demonstrations in challenging settings where the model initially struggles (e.g., MATH level-5).