RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via
  Multi-Stage Reinforcement Learning
Fine-grained visual reasoning remains a core challenge for multimodal large
language models (MLLMs). The recently introduced ReasonMap highlights this gap
by showing that even advanced MLLMs struggle with spatial reasoning in
structured and information-rich settings such as transit maps, a task of clear
practical and scientific importance. However, standard reinforcement learning
(RL) on such tasks is impeded by sparse rewards and unstable optimization. To
address this, we first construct ReasonMap-Plus, an extended dataset that
introduces dense reward signals through Visual Question Answering (VQA) tasks,
enabling effective cold-start training of fine-grained visual understanding
skills. Next, we propose RewardMap, a multi-stage RL framework designed to
improve both visual understanding and reasoning capabilities of MLLMs.
RewardMap incorporates two key designs. First, we introduce a difficulty-aware
reward design that incorporates detail rewards, directly tackling the sparse
rewards while providing richer supervision. Second, we propose a multi-stage RL
scheme that bootstraps training from simple perception to complex reasoning
tasks, offering a more effective cold-start strategy than conventional
Supervised Fine-Tuning (SFT). Experiments on ReasonMap and ReasonMap-Plus
demonstrate that each component of RewardMap contributes to consistent
performance gains, while their combination yields the best results. Moreover,
models trained with RewardMap achieve an average improvement of 3.47% across 6
benchmarks spanning spatial reasoning, fine-grained visual reasoning, and
general tasks beyond transit maps, underscoring enhanced visual understanding
and reasoning capabilities.
68df4813df49fb0df1e03d44
Sicheng Feng
claimed_verified
2025-10-03T12:29:12.265Z
67a4a26d5e65aa63c6d30e68
https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg
Sicheng Feng
user
FSCCS
68df4813df49fb0df1e03d45
Kaiwen Tuo
claimed_verified
2025-10-03T12:29:05.104Z
67c9af33a947280a4afede74
/avatars/23157a9d305aec17564fd229b84a2ca3.svg
Kaiwen TUO
user
KaiwenTUO
68df4813df49fb0df1e03d46
Song Wang
claimed_verified
2025-10-03T12:29:07.845Z
66863d26e2b71e3d09189ae9
/avatars/3c0e6f30e053f2e622ae75e1dc43edba.svg
Song Wang
user
songw-zju
68df4813df49fb0df1e03d47
Lingdong Kong
claimed_verified
2025-10-24T16:15:01.274Z
62df78222d89ce551ce0f71d
/avatars/89fba294cff2d2f941d121c1923e4c76.svg
Lingdong Kong
user
ldkong
68df4813df49fb0df1e03d48
Jianke Zhu
68df4813df49fb0df1e03d49
Huan Wang
claimed_verified
2025-10-06T03:21:18.097Z
62b624f3b52bef716e248fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
Huan Wang
user
Huan-WhoRegisteredMyName
RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via
  Multi-Stage Reinforcement Learning
Fine-grained visual reasoning remains a core challenge for multimodal large
language models (MLLMs). The recently introduced ReasonMap highlights this gap
by showing that even advanced MLLMs struggle with spatial reasoning in
structured and information-rich settings such as transit maps, a task of clear
practical and scientific importance. However, standard reinforcement learning
(RL) on such tasks is impeded by sparse rewards and unstable optimization. To
address this, we first construct ReasonMap-Plus, an extended dataset that
introduces dense reward signals through Visual Question Answering (VQA) tasks,
enabling effective cold-start training of fine-grained visual understanding
skills. Next, we propose RewardMap, a multi-stage RL framework designed to
improve both visual understanding and reasoning capabilities of MLLMs.
RewardMap incorporates two key designs. First, we introduce a difficulty-aware
reward design that incorporates detail rewards, directly tackling the sparse
rewards while providing richer supervision. Second, we propose a multi-stage RL
scheme that bootstraps training from simple perception to complex reasoning
tasks, offering a more effective cold-start strategy than conventional
Supervised Fine-Tuning (SFT). Experiments on ReasonMap and ReasonMap-Plus
demonstrate that each component of RewardMap contributes to consistent
performance gains, while their combination yields the best results. Moreover,
models trained with RewardMap achieve an average improvement of 3.47% across 6
benchmarks spanning spatial reasoning, fine-grained visual reasoning, and
general tasks beyond transit maps, underscoring enhanced visual understanding
and reasoning capabilities.
RewardMap, a multi-stage reinforcement learning framework, enhances multimodal large language models' visual understanding and reasoning skills through dense reward signals and a difficulty-aware reward design.
ReasonMap ReasonMap-Plus Visual Question Answering (VQA) reinforcement learning (RL) sparse rewards unstable optimization difficulty-aware reward design multi-stage RL cold-start training Supervised Fine-Tuning (SFT) spatial reasoning fine-grained visual reasoning