Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust
  Text-based Person Retrieval
Although Contrastive Language-Image Pre-training (CLIP) exhibits strong
performance across diverse vision tasks, its application to person
representation learning faces two critical challenges: (i) the scarcity of
large-scale annotated vision-language data focused on person-centric images,
and (ii) the inherent limitations of global contrastive learning, which
struggles to maintain discriminative local features crucial for fine-grained
matching while remaining vulnerable to noisy text tokens. This work advances
CLIP for person representation learning through synergistic improvements in
data curation and model architecture. First, we develop a noise-resistant data
construction pipeline that leverages the in-context learning capabilities of
MLLMs to automatically filter and caption web-sourced images. This yields
WebPerson, a large-scale dataset of 5M high-quality person-centric image-text
pairs. Second, we introduce the GA-DMS (Gradient-Attention Guided Dual-Masking
Synergetic) framework, which improves cross-modal alignment by adaptively
masking noisy textual tokens based on the gradient-attention similarity score.
Additionally, we incorporate masked token prediction objectives that compel the
model to predict informative text tokens, enhancing fine-grained semantic
representation learning. Extensive experiments show that GA-DMS achieves
state-of-the-art performance across multiple benchmarks.
68c375f3fc1747b91240397f
Tianlu Zheng
68c375f3fc1747b912403980
Yifan Zhang
68c375f3fc1747b912403981
Xiang An
68c375f3fc1747b912403982
Ziyong Feng
68c375f3fc1747b912403983
Kaicheng Yang
claimed_verified
2025-09-12T16:09:24.874Z
63e202f352b7578dba448ab5
https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
Yang
user
Kaichengalex
68c375f3fc1747b912403984
Qichuan Ding
Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust
  Text-based Person Retrieval
Although Contrastive Language-Image Pre-training (CLIP) exhibits strong
performance across diverse vision tasks, its application to person
representation learning faces two critical challenges: (i) the scarcity of
large-scale annotated vision-language data focused on person-centric images,
and (ii) the inherent limitations of global contrastive learning, which
struggles to maintain discriminative local features crucial for fine-grained
matching while remaining vulnerable to noisy text tokens. This work advances
CLIP for person representation learning through synergistic improvements in
data curation and model architecture. First, we develop a noise-resistant data
construction pipeline that leverages the in-context learning capabilities of
MLLMs to automatically filter and caption web-sourced images. This yields
WebPerson, a large-scale dataset of 5M high-quality person-centric image-text
pairs. Second, we introduce the GA-DMS (Gradient-Attention Guided Dual-Masking
Synergetic) framework, which improves cross-modal alignment by adaptively
masking noisy textual tokens based on the gradient-attention similarity score.
Additionally, we incorporate masked token prediction objectives that compel the
model to predict informative text tokens, enhancing fine-grained semantic
representation learning. Extensive experiments show that GA-DMS achieves
state-of-the-art performance across multiple benchmarks.
GA-DMS framework enhances CLIP for person representation learning by improving data quality and model architecture, achieving state-of-the-art performance.
Contrastive Language-Image Pre-training CLIP person representation learning global contrastive learning local features fine-grained matching noise-resistant data construction MLLMs WebPerson GA-DMS Gradient-Attention Guided Dual-Masking Synergetic cross-modal alignment gradient-attention similarity score masked token prediction fine-grained semantic representation learning