Reinforcement Learning for Reasoning in Small LLMs: What Works and What
  Doesn't
Enhancing the reasoning capabilities of large language models (LLMs)
typically relies on massive computational resources and extensive datasets,
limiting accessibility for resource-constrained settings. Our study
investigates the potential of reinforcement learning (RL) to improve reasoning
in small LLMs, focusing on a 1.5-billion-parameter model,
DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA
A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy
Optimization (GRPO) algorithm and curating a compact, high-quality mathematical
reasoning dataset, we conducted three experiments to explore model behavior and
performance. Our results demonstrate rapid reasoning gains - e.g., AMC23
accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing
o1-preview - using only 7,000 samples and a $42 training cost, compared to
thousands of dollars for baseline models. However, challenges such as
optimization instability and length constraints emerged with prolonged
training. These findings highlight the efficacy of RL-based fine-tuning for
small LLMs, offering a cost-effective alternative to large-scale approaches. We
release our code and datasets as open-source resources, providing insights into
trade-offs and laying a foundation for scalable, reasoning-capable LLMs in
resource-limited environments. All are available at
https://github.com/knoveleng/open-rs.
67dd1a9cfa598c90d14e9b47
Quy-Anh Dang
extracted_confirmed
2025-03-21T07:52:47.629Z
645b663eca5d8a297712f2e1
/avatars/c61dcba43feb879088b15b525e441cb9.svg
Quy-Anh Dang
user
quyanh
67dd1a9cfa598c90d14e9b48
Chris Ngo
extracted_pending
2025-03-23T13:42:21.964Z
63105f7463b70252b4775783
/avatars/89cfe71302cde6c2834f58d44bbcdbd5.svg
Chris Ngo
user
tnngo2
Reinforcement Learning for Reasoning in Small LLMs: What Works and What
  Doesn't
Enhancing the reasoning capabilities of large language models (LLMs)
typically relies on massive computational resources and extensive datasets,
limiting accessibility for resource-constrained settings. Our study
investigates the potential of reinforcement learning (RL) to improve reasoning
in small LLMs, focusing on a 1.5-billion-parameter model,
DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA
A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy
Optimization (GRPO) algorithm and curating a compact, high-quality mathematical
reasoning dataset, we conducted three experiments to explore model behavior and
performance. Our results demonstrate rapid reasoning gains - e.g., AMC23
accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing
o1-preview - using only 7,000 samples and a $42 training cost, compared to
thousands of dollars for baseline models. However, challenges such as
optimization instability and length constraints emerged with prolonged
training. These findings highlight the efficacy of RL-based fine-tuning for
small LLMs, offering a cost-effective alternative to large-scale approaches. We
release our code and datasets as open-source resources, providing insights into
trade-offs and laying a foundation for scalable, reasoning-capable LLMs in
resource-limited environments. All are available at
https://github.com/knoveleng/open-rs.
reinforcement learning (RL) Group Relative Policy Optimization (GRPO) mathematical reasoning dataset AMC23 AIME24 optimization instability RL-based fine-tuning scalable reasoning-capable LLMs