Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement
  Learning for General LLM Reasoning
Recent advances in Large Language Models (LLMs) have underscored the
potential of Reinforcement Learning (RL) to facilitate the emergence of
reasoning capabilities. Despite the encouraging results, a fundamental dilemma
persists as RL improvement relies on learning from high-quality samples, yet
the exploration for such samples remains bounded by the inherent limitations of
LLMs. This, in effect, creates an undesirable cycle in which what cannot be
explored cannot be learned. In this work, we propose Rubric-Scaffolded
Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework
designed to break the exploration bottleneck for general LLM reasoning.
Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit
scaffolding for exploration during rollout generation, where different rubrics
are provided as external guidance within task instructions to steer diverse
high-quality responses. This guidance is gradually decayed over time,
encouraging the model to internalize the underlying reasoning patterns; (2)
verifiable rewards for exploitation during model training, where we can obtain
robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL
on general reasoning tasks. Extensive experiments demonstrate the superiority
of the proposed RuscaRL across various benchmarks, effectively expanding
reasoning boundaries under the best-of-N evaluation. Notably, RuscaRL
significantly boosts Qwen-2.5-7B-Instruct from 23.6 to 50.3 on HealthBench-500,
surpassing GPT-4.1. Furthermore, our fine-tuned variant on
Qwen3-30B-A3B-Instruct achieves 61.1 on HealthBench-500, outperforming leading
LLMs including OpenAI-o3.
68ad2b3c86b21a0e2e358de3
Yang Zhou
claimed_verified
2025-08-26T09:49:31.033Z
670ff71f6b8de497472a81dc
https://cdn-avatars.huggingface.co/v1/production/uploads/670ff71f6b8de497472a81dc/R2ydPAk6GU5Sk81Qja7E_.jpeg
YANG ZHOU
user
BAOLONGZHANSHEN
68ad2b3c86b21a0e2e358de4
Sunzhu Li
68ad2b3c86b21a0e2e358de5
Shunyu Liu
68ad2b3c86b21a0e2e358de6
Wenkai Fang
68ad2b3c86b21a0e2e358de7
Jiale Zhao
68ad2b3c86b21a0e2e358de8
Jingwen Yang
68ad2b3c86b21a0e2e358de9
Jianwei Lv
68ad2b3c86b21a0e2e358dea
Kongcheng Zhang
68ad2b3c86b21a0e2e358deb
Yihe Zhou
68ad2b3c86b21a0e2e358dec
Hengtong Lu
68ad2b3c86b21a0e2e358ded
Wei Chen
68ad2b3c86b21a0e2e358dee
Yan Xie
68ad2b3c86b21a0e2e358def
Mingli Song
Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement
  Learning for General LLM Reasoning
Recent advances in Large Language Models (LLMs) have underscored the
potential of Reinforcement Learning (RL) to facilitate the emergence of
reasoning capabilities. Despite the encouraging results, a fundamental dilemma
persists as RL improvement relies on learning from high-quality samples, yet
the exploration for such samples remains bounded by the inherent limitations of
LLMs. This, in effect, creates an undesirable cycle in which what cannot be
explored cannot be learned. In this work, we propose Rubric-Scaffolded
Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework
designed to break the exploration bottleneck for general LLM reasoning.
Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit
scaffolding for exploration during rollout generation, where different rubrics
are provided as external guidance within task instructions to steer diverse
high-quality responses. This guidance is gradually decayed over time,
encouraging the model to internalize the underlying reasoning patterns; (2)
verifiable rewards for exploitation during model training, where we can obtain
robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL
on general reasoning tasks. Extensive experiments demonstrate the superiority
of the proposed RuscaRL across various benchmarks, effectively expanding
reasoning boundaries under the best-of-N evaluation. Notably, RuscaRL
significantly boosts Qwen-2.5-7B-Instruct from 23.6 to 50.3 on HealthBench-500,
surpassing GPT-4.1. Furthermore, our fine-tuned variant on
Qwen3-30B-A3B-Instruct achieves 61.1 on HealthBench-500, outperforming leading
LLMs including OpenAI-o3.
RuscaRL, a novel instructional scaffolding framework, enhances LLM reasoning by using rubrics for exploration and verifiable rewards, significantly improving performance on reasoning tasks.
Large Language Models Reinforcement Learning Rubric-Scaffolded Reinforcement Learning RuscaRL checklist-style rubrics rollout generation LLM-as-a-Judge best-of-N evaluation HealthBench-500 Qwen-2.5-7B-Instruct GPT-4.1 Qwen3-30B-A3B-Instruct OpenAI-o3