Detect Anything via Next Point Prediction
Object detection has long been dominated by traditional coordinate
regression-based models, such as YOLO, DETR, and Grounding DINO. Although
recent efforts have attempted to leverage MLLMs to tackle this task, they face
challenges like low recall rate, duplicate predictions, coordinate
misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a
3B-scale MLLM that achieves state-of-the-art object perception performance. On
benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or
exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot
setting. This is enabled by three key designs: 1) Task Formulation: we use
special tokens to represent quantized coordinates from 0 to 999, reducing the
model's learning difficulty and improving token efficiency for coordinate
prediction; 2) Data Engines: we construct multiple data engines to generate
high-quality grounding, referring, and pointing data, providing semantically
rich supervision for training; \3) Training Pipelines: we employ a two-stage
training process, combining supervised fine-tuning on 22 million data with
GRPO-based reinforcement post-training. This RL post-training leverages
geometry-aware rewards to effectively bridge the discrete-to-continuous
coordinate prediction gap, improve box accuracy, and mitigate undesirable
behaviors like duplicate predictions that stem from the teacher-guided nature
of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent
language understanding enables versatile capabilities such as object referring,
pointing, visual prompting, GUI grounding, spatial referring, OCR and
key-pointing, all systematically evaluated on dedicated benchmarks. We believe
that Rex-Omni paves the way for more versatile and language-aware visual
perception systems.
68eefe5c486b78128f0e337d
Qing Jiang
68eefe5c486b78128f0e337e
Junan Huo
68eefe5c486b78128f0e337f
Xingyu Chen
68eefe5c486b78128f0e3380
Yuda Xiong
68eefe5c486b78128f0e3381
Zhaoyang Zeng
68eefe5c486b78128f0e3382
Yihao Chen
68eefe5c486b78128f0e3383
Tianhe Ren
68eefe5c486b78128f0e3384
Junzhi Yu
68eefe5c486b78128f0e3385
Lei Zhang
Detect Anything via Next Point Prediction
Object detection has long been dominated by traditional coordinate
regression-based models, such as YOLO, DETR, and Grounding DINO. Although
recent efforts have attempted to leverage MLLMs to tackle this task, they face
challenges like low recall rate, duplicate predictions, coordinate
misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a
3B-scale MLLM that achieves state-of-the-art object perception performance. On
benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or
exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot
setting. This is enabled by three key designs: 1) Task Formulation: we use
special tokens to represent quantized coordinates from 0 to 999, reducing the
model's learning difficulty and improving token efficiency for coordinate
prediction; 2) Data Engines: we construct multiple data engines to generate
high-quality grounding, referring, and pointing data, providing semantically
rich supervision for training; \3) Training Pipelines: we employ a two-stage
training process, combining supervised fine-tuning on 22 million data with
GRPO-based reinforcement post-training. This RL post-training leverages
geometry-aware rewards to effectively bridge the discrete-to-continuous
coordinate prediction gap, improve box accuracy, and mitigate undesirable
behaviors like duplicate predictions that stem from the teacher-guided nature
of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent
language understanding enables versatile capabilities such as object referring,
pointing, visual prompting, GUI grounding, spatial referring, OCR and
key-pointing, all systematically evaluated on dedicated benchmarks. We believe
that Rex-Omni paves the way for more versatile and language-aware visual
perception systems.