Referring to Any Person
Humans are undoubtedly the most important participants in computer vision,
and the ability to detect any individual given a natural language description,
a task we define as referring to any person, holds substantial practical value.
However, we find that existing models generally fail to achieve real-world
usability, and current benchmarks are limited by their focus on one-to-one
referring, that hinder progress in this area. In this work, we revisit this
task from three critical perspectives: task definition, dataset design, and
model architecture. We first identify five aspects of referable entities and
three distinctive characteristics of this task. Next, we introduce HumanRef, a
novel dataset designed to tackle these challenges and better reflect real-world
applications. From a model design perspective, we integrate a multimodal large
language model with an object detection framework, constructing a robust
referring model named RexSeek. Experimental results reveal that
state-of-the-art models, which perform well on commonly used benchmarks like
RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple
individuals. In contrast, RexSeek not only excels in human referring but also
generalizes effectively to common object referring, making it broadly
applicable across various perception tasks. Code is available at
https://github.com/IDEA-Research/RexSeek
67d15293e3afecf451aceab7
Qing Jiang
67d15293e3afecf451aceab8
Lin Wu
67d15293e3afecf451aceab9
Zhaoyang Zeng
67d15293e3afecf451aceaba
Tianhe Ren
67d15293e3afecf451aceabb
Yuda Xiong
67d15293e3afecf451aceabc
Yihao Chen
67d15293e3afecf451aceabd
Qin Liu
67d15293e3afecf451aceabe
Lei Zhang
Referring to Any Person
Humans are undoubtedly the most important participants in computer vision,
and the ability to detect any individual given a natural language description,
a task we define as referring to any person, holds substantial practical value.
However, we find that existing models generally fail to achieve real-world
usability, and current benchmarks are limited by their focus on one-to-one
referring, that hinder progress in this area. In this work, we revisit this
task from three critical perspectives: task definition, dataset design, and
model architecture. We first identify five aspects of referable entities and
three distinctive characteristics of this task. Next, we introduce HumanRef, a
novel dataset designed to tackle these challenges and better reflect real-world
applications. From a model design perspective, we integrate a multimodal large
language model with an object detection framework, constructing a robust
referring model named RexSeek. Experimental results reveal that
state-of-the-art models, which perform well on commonly used benchmarks like
RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple
individuals. In contrast, RexSeek not only excels in human referring but also
generalizes effectively to common object referring, making it broadly
applicable across various perception tasks. Code is available at
https://github.com/IDEA-Research/RexSeek
A novel dataset and model integrating a multimodal large language model with an object detection framework improve human referring and generalize to common object referring tasks.
multimodal large language model object detection framework referable entities HumanRef RexSeek RefCOCO/+/g