Progressive Language-guided Visual Learning for Multi-Task Visual
  Grounding
Multi-task visual grounding (MTVG) includes two sub-tasks, i.e., Referring
Expression Comprehension (REC) and Referring Expression Segmentation (RES). The
existing representative approaches generally follow the research pipeline which
mainly consists of three core procedures, including independent feature
extraction for visual and linguistic modalities, respectively, cross-modal
interaction module, and independent prediction heads for different sub-tasks.
Albeit achieving remarkable performance, this research line has two
limitations: 1) The linguistic content has not been fully injected into the
entire visual backbone for boosting more effective visual feature extraction
and it needs an extra cross-modal interaction module; 2) The relationship
between REC and RES tasks is not effectively exploited to help the
collaborative prediction for more accurate output. To deal with these problems,
in this paper, we propose a Progressive Language-guided Visual Learning
framework for multi-task visual grounding, called PLVL, which not only finely
mine the inherent feature expression of the visual modality itself but also
progressively inject the language information to help learn linguistic-related
visual features. In this manner, our PLVL does not need additional cross-modal
fusion module while fully introducing the language guidance. Furthermore, we
analyze that the localization center for REC would help identify the
to-be-segmented object region for RES to some extent. Inspired by this
investigation, we design a multi-task head to accomplish collaborative
predictions for these two sub-tasks. Extensive experiments conducted on several
benchmark datasets comprehensively substantiate that our PLVL obviously
outperforms the representative methods in both REC and RES tasks.
https://github.com/jcwang0602/PLVL
680a164533e15279df86adec
Jingchao Wang
claimed_verified
2025-04-24T11:25:46.213Z
65eaa07cb6c760d77468b4b6
/avatars/4a1aae58986b40444351e0a167ca807c.svg
Jingchao Wang
user
jcwang0602
680a164533e15279df86aded
Hong Wang
680a164533e15279df86adee
Wenlong Zhang
680a164533e15279df86adef
Kunhua Ji
680a164533e15279df86adf0
Dingjiang Huang
680a164533e15279df86adf1
Yefeng Zheng
Progressive Language-guided Visual Learning for Multi-Task Visual
  Grounding
Multi-task visual grounding (MTVG) includes two sub-tasks, i.e., Referring
Expression Comprehension (REC) and Referring Expression Segmentation (RES). The
existing representative approaches generally follow the research pipeline which
mainly consists of three core procedures, including independent feature
extraction for visual and linguistic modalities, respectively, cross-modal
interaction module, and independent prediction heads for different sub-tasks.
Albeit achieving remarkable performance, this research line has two
limitations: 1) The linguistic content has not been fully injected into the
entire visual backbone for boosting more effective visual feature extraction
and it needs an extra cross-modal interaction module; 2) The relationship
between REC and RES tasks is not effectively exploited to help the
collaborative prediction for more accurate output. To deal with these problems,
in this paper, we propose a Progressive Language-guided Visual Learning
framework for multi-task visual grounding, called PLVL, which not only finely
mine the inherent feature expression of the visual modality itself but also
progressively inject the language information to help learn linguistic-related
visual features. In this manner, our PLVL does not need additional cross-modal
fusion module while fully introducing the language guidance. Furthermore, we
analyze that the localization center for REC would help identify the
to-be-segmented object region for RES to some extent. Inspired by this
investigation, we design a multi-task head to accomplish collaborative
predictions for these two sub-tasks. Extensive experiments conducted on several
benchmark datasets comprehensively substantiate that our PLVL obviously
outperforms the representative methods in both REC and RES tasks.
https://github.com/jcwang0602/PLVL
A Progressive Language-guided Visual Learning framework for multi-task visual grounding that integrates linguistic information into the visual backbone and leverages shared features between sub-tasks to improve performance in both Referring Expression Comprehension and Referring Expression Segmentation.
Multi-task visual grounding Referring Expression Comprehension Referring Expression Segmentation independent feature extraction cross-modal interaction module independent prediction heads visual backbone language information linguistic-related visual features multi-task head shared features benchmark datasets