RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic
  Sampling
Rule-based reasoning has been acknowledged as one of the fundamental problems
in reasoning, while deviations in rule formats, types, and complexity in
real-world applications pose severe challenges. Recent studies have shown that
large reasoning models (LRMs) have remarkable reasoning capabilities, and their
performance is substantially enhanced by reinforcement learning (RL). However,
it remains an open question whether small reasoning models (SRMs) can learn
rule-based reasoning effectively with robust generalization across diverse
tasks and domains. To address this, we introduce Reinforced Rule-based
Reasoning, a.k.a. RuleReasoner, a simple yet effective method to conduct
rule-based reasoning via a wide collection of curated tasks and a novel
domain-aware dynamic sampling approach. Specifically, RuleReasoner resamples
each training batch by updating the sampling weights of different domains based
on historical rewards. This facilitates domain augmentation and flexible online
learning schedules for RL, obviating the need for pre-hoc human-engineered
mix-training recipes used in existing methods. Empirical evaluations on
in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that
RuleReasoner outperforms frontier LRMs by a significant margin (Delta4.1%
average points on eight ID tasks and Delta10.4% average points on three OOD
tasks over OpenAI-o1). Notably, our approach also exhibits higher computational
efficiency compared to prior dynamic sampling methods for RL.
684936e842e4f9106973f45e
Yang Liu
claimed_verified
2025-06-24T13:33:33.215Z
6191cc9e6d34e827404cebab
https://cdn-avatars.huggingface.co/v1/production/uploads/1674119843175-6191cc9e6d34e827404cebab.jpeg
Yang
user
jacklanda
684936e842e4f9106973f45f
Jiaqi Li
684936e842e4f9106973f460
Zilong Zheng
extracted_pending
2025-06-11T07:57:29.119Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic
  Sampling
Rule-based reasoning has been acknowledged as one of the fundamental problems
in reasoning, while deviations in rule formats, types, and complexity in
real-world applications pose severe challenges. Recent studies have shown that
large reasoning models (LRMs) have remarkable reasoning capabilities, and their
performance is substantially enhanced by reinforcement learning (RL). However,
it remains an open question whether small reasoning models (SRMs) can learn
rule-based reasoning effectively with robust generalization across diverse
tasks and domains. To address this, we introduce Reinforced Rule-based
Reasoning, a.k.a. RuleReasoner, a simple yet effective method to conduct
rule-based reasoning via a wide collection of curated tasks and a novel
domain-aware dynamic sampling approach. Specifically, RuleReasoner resamples
each training batch by updating the sampling weights of different domains based
on historical rewards. This facilitates domain augmentation and flexible online
learning schedules for RL, obviating the need for pre-hoc human-engineered
mix-training recipes used in existing methods. Empirical evaluations on
in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that
RuleReasoner outperforms frontier LRMs by a significant margin (Delta4.1%
average points on eight ID tasks and Delta10.4% average points on three OOD
tasks over OpenAI-o1). Notably, our approach also exhibits higher computational
efficiency compared to prior dynamic sampling methods for RL.
RuleReasoner enhances rule-based reasoning in small models through dynamic domain sampling, achieving superior performance and efficiency compared to large models.
reinforcement learning rule-based reasoning large reasoning models small reasoning models domain-aware dynamic sampling historical rewards in-distribution out-of-distribution computational efficiency