Active-O3: Empowering Multimodal Large Language Models with Active
  Perception via GRPO
Active vision, also known as active perception, refers to the process of
actively selecting where and how to look in order to gather task-relevant
information. It is a critical component of efficient perception and
decision-making in humans and advanced embodied agents. Recently, the use of
Multimodal Large Language Models (MLLMs) as central planning and
decision-making modules in robotic systems has gained extensive attention.
However, despite the importance of active perception in embodied intelligence,
there is little to no exploration of how MLLMs can be equipped with or learn
active perception capabilities. In this paper, we first provide a systematic
definition of MLLM-based active perception tasks. We point out that the
recently proposed GPT-o3 model's zoom-in search strategy can be regarded as a
special case of active perception; however, it still suffers from low search
efficiency and inaccurate region selection. To address these issues, we propose
ACTIVE-O3, a purely reinforcement learning based training framework built on
top of GRPO, designed to equip MLLMs with active perception capabilities. We
further establish a comprehensive benchmark suite to evaluate ACTIVE-O3 across
both general open-world tasks, such as small-object and dense object grounding,
and domain-specific scenarios, including small object detection in remote
sensing and autonomous driving, as well as fine-grained interactive
segmentation. In addition, ACTIVE-O3 also demonstrates strong zero-shot
reasoning abilities on the V* Benchmark, without relying on any explicit
reasoning data. We hope that our work can provide a simple codebase and
evaluation protocol to facilitate future research on active perception in
MLLMs.
6836711b80ed824b28f7a78a
Muzhi Zhu
claimed_verified
2025-05-28T08:57:55.082Z
632179745fc60c44fd91fc33
/avatars/37d4fefbcc19f091dccffefec9706de2.svg
zhumuzhi
user
Z-MU-Z
6836711b80ed824b28f7a78b
Hao Zhong
6836711b80ed824b28f7a78c
Canyu Zhao
claimed_verified
2025-05-28T08:57:48.936Z
646efd223dd912a539e0bd46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
Canyu Zhao
user
Canyu
6836711b80ed824b28f7a78d
Zongze Du
6836711b80ed824b28f7a78e
Zheng Huang
6836711b80ed824b28f7a78f
Mingyu Liu
claimed_verified
2025-05-28T08:57:51.293Z
652e25d2e647b0ee0a024f26
/avatars/b5c65cf6c8d0ddc9b8ef0226e0295d56.svg
Mingyu Liu
user
MingyuLiu
6836711b80ed824b28f7a790
Hao Chen
6836711b80ed824b28f7a791
Cheng Zou
6836711b80ed824b28f7a792
Jingdong Chen
6836711b80ed824b28f7a793
Ming Yang
6836711b80ed824b28f7a794
Chunhua Shen
Active-O3: Empowering Multimodal Large Language Models with Active
  Perception via GRPO
Active vision, also known as active perception, refers to the process of
actively selecting where and how to look in order to gather task-relevant
information. It is a critical component of efficient perception and
decision-making in humans and advanced embodied agents. Recently, the use of
Multimodal Large Language Models (MLLMs) as central planning and
decision-making modules in robotic systems has gained extensive attention.
However, despite the importance of active perception in embodied intelligence,
there is little to no exploration of how MLLMs can be equipped with or learn
active perception capabilities. In this paper, we first provide a systematic
definition of MLLM-based active perception tasks. We point out that the
recently proposed GPT-o3 model's zoom-in search strategy can be regarded as a
special case of active perception; however, it still suffers from low search
efficiency and inaccurate region selection. To address these issues, we propose
ACTIVE-O3, a purely reinforcement learning based training framework built on
top of GRPO, designed to equip MLLMs with active perception capabilities. We
further establish a comprehensive benchmark suite to evaluate ACTIVE-O3 across
both general open-world tasks, such as small-object and dense object grounding,
and domain-specific scenarios, including small object detection in remote
sensing and autonomous driving, as well as fine-grained interactive
segmentation. In addition, ACTIVE-O3 also demonstrates strong zero-shot
reasoning abilities on the V* Benchmark, without relying on any explicit
reasoning data. We hope that our work can provide a simple codebase and
evaluation protocol to facilitate future research on active perception in
MLLMs.
A reinforcement learning framework, ACTIVE-O3, is proposed to equip Multimodal Large Language Models with active perception capabilities and tested across various tasks and benchmarks.
Multimodal Large Language Models MLLMs active perception GPT-o3 zoom-in search ACTIVE-O3 reinforcement learning GRPO small-object grounding dense object grounding small object detection remote sensing autonomous driving fine-grained interactive segmentation V* Benchmark zero-shot reasoning