Catching the Details: Self-Distilled RoI Predictors for Fine-Grained
  MLLM Perception
Multimodal Large Language Models (MLLMs) require high-resolution visual
information to perform fine-grained perception, yet processing entire
high-resolution images is computationally prohibitive. While recent methods
leverage a Region-of-Interest (RoI) mechanism to focus on salient areas, they
typically present a difficult trade-off: training-based approaches depend on
large-scale annotated datasets, while training-free methods that utilize the
model's internal attention are computationally inefficient and less accurate,
requiring either multi-pass prefill stages or reliance on the slow
auto-regressive decoding process. In this paper, we propose an efficient,
annotation-free Self-Distilled Region Proposal Network (SD-RPN) that resolves
this trade-off. The SD-RPN is built around a pipeline that transforms the noisy
attention maps from the MLLM's middle layers into high-quality pseudo-RoI
labels by explicitly denoising the signal and resolving ambiguity. We use these
labels to train a lightweight Region Proposal Network (RPN) that learns a more
precise localization. This RPN is also highly efficient, predicting the RoI in
a single forward pass using features from the MLLM's middle layers, decoupling
RoI identification from the auto-regressive generation and avoiding costly
multi-pass operations.To validate our approach, we integrate the framework into
the LLaVA-1.5 architecture. Despite being trained on only a few (e.g. 10K)
question-answer pairs, our method demonstrates exceptional data efficiency and
generalization, achieving over a 10% absolute accuracy improvement on unseen
benchmarks, including TextVQA, DocVQA, and V-Star. Our work presents a
practical and scalable solution for enhancing the fine-grained perception of
MLLMs without requiring costly supervision or full model fine-tuning. Code is
available at https://github.com/YuHengsss/SD-RPN.
68ddc31a6024653e8a3ed03b
Yuheng Shi
claimed_verified
2025-10-02T03:08:10.975Z
65d6c252ca16ef9ba7e76664
/avatars/45d6e8bdd7b944e912f6674696a941b3.svg
Yuheng Shi
user
YuhengSSS
68ddc31a6024653e8a3ed03c
Xiaohuan Pei
68ddc31a6024653e8a3ed03d
Minjing Dong
68ddc31a6024653e8a3ed03e
Chang Xu
Catching the Details: Self-Distilled RoI Predictors for Fine-Grained
  MLLM Perception
Multimodal Large Language Models (MLLMs) require high-resolution visual
information to perform fine-grained perception, yet processing entire
high-resolution images is computationally prohibitive. While recent methods
leverage a Region-of-Interest (RoI) mechanism to focus on salient areas, they
typically present a difficult trade-off: training-based approaches depend on
large-scale annotated datasets, while training-free methods that utilize the
model's internal attention are computationally inefficient and less accurate,
requiring either multi-pass prefill stages or reliance on the slow
auto-regressive decoding process. In this paper, we propose an efficient,
annotation-free Self-Distilled Region Proposal Network (SD-RPN) that resolves
this trade-off. The SD-RPN is built around a pipeline that transforms the noisy
attention maps from the MLLM's middle layers into high-quality pseudo-RoI
labels by explicitly denoising the signal and resolving ambiguity. We use these
labels to train a lightweight Region Proposal Network (RPN) that learns a more
precise localization. This RPN is also highly efficient, predicting the RoI in
a single forward pass using features from the MLLM's middle layers, decoupling
RoI identification from the auto-regressive generation and avoiding costly
multi-pass operations.To validate our approach, we integrate the framework into
the LLaVA-1.5 architecture. Despite being trained on only a few (e.g. 10K)
question-answer pairs, our method demonstrates exceptional data efficiency and
generalization, achieving over a 10% absolute accuracy improvement on unseen
benchmarks, including TextVQA, DocVQA, and V-Star. Our work presents a
practical and scalable solution for enhancing the fine-grained perception of
MLLMs without requiring costly supervision or full model fine-tuning. Code is
available at https://github.com/YuHengsss/SD-RPN.
A self-distilled Region Proposal Network (SD-RPN) enhances the fine-grained perception of Multimodal Large Language Models (MLLMs) by efficiently generating high-quality Region-of-Interest (RoI) labels without large-scale annotated datasets.
Multimodal Large Language Models Region-of-Interest RoI Region Proposal Network RPN self-distilled denoising attention maps LLaVA-1.5 TextVQA DocVQA V-Star