QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning
  for LLMs
We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for
large language models (LLMs). While RL is essential for LLMs' reasoning
capabilities, it is resource-intensive, requiring substantial GPU memory and
long rollout durations. QeRL addresses these issues by combining NVFP4
quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL
while reducing memory overhead. Beyond efficiency, our findings show that
quantization noise increases policy entropy, enhancing exploration, and
enabling the discovery of better strategies during RL. To further optimize
exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism,
which dynamically adjusts noise during training. Experiments demonstrate that
QeRL delivers over 1.5 times speedup in the rollout phase. Moreover, this is
the first framework to enable RL training of a 32B LLM on a single H100 80GB
GPU, while delivering overall speedups for RL training. It also achieves faster
reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while
matching the performance of full-parameter fine-tuning on mathematical
benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These
results establish QeRL as an efficient and effective framework for RL training
in LLMs.
68edb7abde1fee572713a7df
Wei Huang
claimed_verified
2025-10-23T15:02:23.711Z
656db3f53dc1d277e5a64410
https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
Wei Huang
user
AaronHuangWei
68edb7abde1fee572713a7e0
Yi Ge
claimed_verified
2025-10-14T10:23:44.636Z
6698c7f6743d55b416bdc008
https://cdn-avatars.huggingface.co/v1/production/uploads/6698c7f6743d55b416bdc008/pU6qbo6V4PYAqtMFOhrRm.jpeg
Ge Yi
user
GY2233
68edb7abde1fee572713a7e1
Shuai Yang
68edb7abde1fee572713a7e2
Yicheng Xiao
68edb7abde1fee572713a7e3
Huizi Mao
68edb7abde1fee572713a7e4
Yujun Lin
68edb7abde1fee572713a7e5
Hanrong Ye
68edb7abde1fee572713a7e6
Sifei Liu
68edb7abde1fee572713a7e7
Ka Chun Cheung
68edb7abde1fee572713a7e8
Hongxu Yin
68edb7abde1fee572713a7e9
Yao Lu
68edb7abde1fee572713a7ea
Xiaojuan Qi
68edb7abde1fee572713a7eb
Song Han
68edb7abde1fee572713a7ec
Yukang Chen
QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning
  for LLMs
We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for
large language models (LLMs). While RL is essential for LLMs' reasoning
capabilities, it is resource-intensive, requiring substantial GPU memory and
long rollout durations. QeRL addresses these issues by combining NVFP4
quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL
while reducing memory overhead. Beyond efficiency, our findings show that
quantization noise increases policy entropy, enhancing exploration, and
enabling the discovery of better strategies during RL. To further optimize
exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism,
which dynamically adjusts noise during training. Experiments demonstrate that
QeRL delivers over 1.5 times speedup in the rollout phase. Moreover, this is
the first framework to enable RL training of a 32B LLM on a single H100 80GB
GPU, while delivering overall speedups for RL training. It also achieves faster
reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while
matching the performance of full-parameter fine-tuning on mathematical
benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These
results establish QeRL as an efficient and effective framework for RL training
in LLMs.
QeRL, a quantization-enhanced reinforcement learning framework, accelerates RL training for large language models by combining NVFP4 quantization with Low-Rank Adaptation and an Adaptive Quantization Noise mechanism, achieving significant speedups and improved performance.
NVFP4 quantization Low-Rank Adaptation (LoRA) Adaptive Quantization Noise (AQN) reinforcement learning large language models (LLMs) rollout phase policy entropy exploration reward growth GSM8K MATH 500