SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement
  Learning for LLM Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective
for training large language models (LLMs) on complex reasoning tasks, such as
mathematical problem solving. A prerequisite for the scalability of RLVR is a
high-quality problem set with precise and verifiable answers. However, the
scarcity of well-crafted human-labeled math problems and limited-verification
answers in existing distillation-oriented synthetic datasets limit their
effectiveness in RL. Additionally, most problem synthesis strategies
indiscriminately expand the problem set without considering the model's
capabilities, leading to low efficiency in generating useful questions. To
mitigate this issue, we introduce a Self-aware Weakness-driven problem
Synthesis framework (SwS) that systematically identifies model deficiencies and
leverages them for problem augmentation. Specifically, we define weaknesses as
questions that the model consistently fails to learn through its iterative
sampling during RL training. We then extract the core concepts from these
failure cases and synthesize new problems to strengthen the model's weak areas
in subsequent augmented training, enabling it to focus on and gradually
overcome its weaknesses. Without relying on external knowledge distillation,
our framework enables robust generalization byempowering the model to
self-identify and address its weaknesses in RL, yielding average performance
gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning
benchmarks.
684d3eb33b733ba333687407
Xiao Liang
claimed_verified
2025-06-16T12:56:19.750Z
6560763e152b659e623865ae
https://cdn-avatars.huggingface.co/v1/production/uploads/6560763e152b659e623865ae/cTT2jGnPU_8XMrUTvqZ2h.jpeg
Xiao Liang
user
MasterVito
684d3eb33b733ba333687408
Zhong-Zhi Li
claimed_verified
2025-06-16T12:56:21.547Z
6545f575650796837446afca
/avatars/df9fab381e5d43af753b8fc2c7d35865.svg
Zhongzhi Li
user
lizhongzhi2022
684d3eb33b733ba333687409
Yeyun Gong
684d3eb33b733ba33368740a
Yang Wang
claimed_verified
2025-06-18T12:17:07.396Z
604714a0c82d59b7347b55ae
https://cdn-avatars.huggingface.co/v1/production/uploads/604714a0c82d59b7347b55ae/WZFKDDUPi8JS0BK8t7mIv.jpeg
YangWang92
user
yangwang92
684d3eb33b733ba33368740b
Hengyuan Zhang
684d3eb33b733ba33368740c
Yelong Shen
extracted_pending
2025-06-14T09:19:48.976Z
643f615aa16cd6d1f4c581de
/avatars/47753a3e82b44f81881600c52e1e8495.svg
Yeyun Gong
user
yegong
684d3eb33b733ba33368740d
Ying Nian Wu
684d3eb33b733ba33368740e
Weizhu Chen
SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement
  Learning for LLM Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective
for training large language models (LLMs) on complex reasoning tasks, such as
mathematical problem solving. A prerequisite for the scalability of RLVR is a
high-quality problem set with precise and verifiable answers. However, the
scarcity of well-crafted human-labeled math problems and limited-verification
answers in existing distillation-oriented synthetic datasets limit their
effectiveness in RL. Additionally, most problem synthesis strategies
indiscriminately expand the problem set without considering the model's
capabilities, leading to low efficiency in generating useful questions. To
mitigate this issue, we introduce a Self-aware Weakness-driven problem
Synthesis framework (SwS) that systematically identifies model deficiencies and
leverages them for problem augmentation. Specifically, we define weaknesses as
questions that the model consistently fails to learn through its iterative
sampling during RL training. We then extract the core concepts from these
failure cases and synthesize new problems to strengthen the model's weak areas
in subsequent augmented training, enabling it to focus on and gradually
overcome its weaknesses. Without relying on external knowledge distillation,
our framework enables robust generalization byempowering the model to
self-identify and address its weaknesses in RL, yielding average performance
gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning
benchmarks.
A self-aware problem synthesis framework that leverages model weaknesses enhances reinforcement learning with verifiable rewards, improving large language model performance on reasoning tasks.
Reinforcement Learning with Verifiable Rewards RLVR large language models LLMs complex reasoning tasks mathematical problem solving problem set verifiable answers human-labeled math problems synthetic datasets problem synthesis model capabilities self-aware weakness-driven problem augmentation iterative sampling core concepts synthesizing new problems robust generalization reasoning benchmarks