DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
Jinyoung Park, Jeehye Na, Jinyoung Kim, Hyunwoo Kim
Recent works have demonstrated the effectiveness of reinforcement learning (RL)-based post-training for enhancing the reasoning capabilities of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) has shown impressive success using a PPO-style reinforcement algorithm with group-based normalized rewards. However, GRPO has been less explored in Video Large Language Models (VideoLLMs). In this paper, we explore GRPO and identify two problems that deteriorate the effective learning: (1) reliance on safeguards, and (2) vanishing advantage. To mitigate these challenges, we propose DeepVideo-R1, a video large language model trained with Reg-GRPO (Regressive GRPO) and difficulty-aware data augmentation. Reg-GRPO reformulates the GRPO loss function into a regression task that directly predicts the advantage in GRPO, eliminating the need for heuristic safeguards such as the clipping and min functions. It aligns VideoLLMs with advantages, providing effective guidance. The difficulty-aware data augmentation strategy augments input prompts/videos to generate samples at solvable difficulty levels, enabling diverse reward signals. Experimental results show that our approach significantly improves video reasoning performance across multiple benchmarks. Our codes are included in the supplement.