Task Preference Optimization: Improving Multimodal Large Language Models
  with Vision Task Alignment
Current multimodal large language models (MLLMs) struggle with fine-grained
or precise understanding of visuals though they give comprehensive perception
and reasoning in a spectrum of vision applications. Recent studies either
develop tool-using or unify specific visual tasks into the autoregressive
framework, often at the expense of overall multimodal performance. To address
this issue and enhance MLLMs with visual tasks in a scalable fashion, we
propose Task Preference Optimization (TPO), a novel method that utilizes
differentiable task preferences derived from typical fine-grained visual tasks.
TPO introduces learnable task tokens that establish connections between
multiple task-specific heads and the MLLM. By leveraging rich visual labels
during training, TPO significantly enhances the MLLM's multimodal capabilities
and task-specific performance. Through multi-task co-training within TPO, we
observe synergistic benefits that elevate individual task performance beyond
what is achievable through single-task training methodologies. Our
instantiation of this approach with VideoChat and LLaVA demonstrates an overall
14.6% improvement in multimodal performance compared to baseline models.
Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across
various tasks, performing comparably to state-of-the-art supervised models. The
code will be released at https://github.com/OpenGVLab/TPO
67722817633a6043c33212aa
Ziang Yan
admin_assigned
2024-12-30T20:44:33.925Z
65499e5f2a292b3e2e5715a3
/avatars/087b3e36dfb66e044265b856bab31657.svg
ziang yan
user
Aurorana
67722817633a6043c33212ab
Zhilin Li
admin_assigned
2024-12-30T20:44:41.063Z
64412af71a80f6d83cb03318
/avatars/113ad7064e7230eff3ae86ad24c88818.svg
zhilin li
user
white213
67722817633a6043c33212ac
Yinan He
admin_assigned
2024-12-30T20:44:47.500Z
65b9d9961fe588f824fde191
/avatars/a9245958cc998a4b4b870bf2490fdaee.svg
Yinan He
user
yinanhe
67722817633a6043c33212ad
Chenting Wang
67722817633a6043c33212ae
Kunchang Li
admin_assigned
2024-12-30T20:45:06.653Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
67722817633a6043c33212af
Xinhao Li
admin_assigned
2024-12-30T20:45:14.329Z
672f8a28c53c174f39b08ac1
/avatars/9d865f757667de14381d7c4d7ba7e4c4.svg
XINHAO LI
user
xinhaoli
67722817633a6043c33212b0
Xiangyu Zeng
67722817633a6043c33212b1
Zilei Wang
67722817633a6043c33212b2
Yali Wang
67722817633a6043c33212b3
Yu Qiao
67722817633a6043c33212b4
Limin Wang
67722817633a6043c33212b5
Yi Wang
Task Preference Optimization: Improving Multimodal Large Language Models
  with Vision Task Alignment
Current multimodal large language models (MLLMs) struggle with fine-grained
or precise understanding of visuals though they give comprehensive perception
and reasoning in a spectrum of vision applications. Recent studies either
develop tool-using or unify specific visual tasks into the autoregressive
framework, often at the expense of overall multimodal performance. To address
this issue and enhance MLLMs with visual tasks in a scalable fashion, we
propose Task Preference Optimization (TPO), a novel method that utilizes
differentiable task preferences derived from typical fine-grained visual tasks.
TPO introduces learnable task tokens that establish connections between
multiple task-specific heads and the MLLM. By leveraging rich visual labels
during training, TPO significantly enhances the MLLM's multimodal capabilities
and task-specific performance. Through multi-task co-training within TPO, we
observe synergistic benefits that elevate individual task performance beyond
what is achievable through single-task training methodologies. Our
instantiation of this approach with VideoChat and LLaVA demonstrates an overall
14.6% improvement in multimodal performance compared to baseline models.
Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across
various tasks, performing comparably to state-of-the-art supervised models. The
code will be released at https://github.com/OpenGVLab/TPO
Task Preference Optimization (TPO) enhances multimodal large language models with fine-grained visual tasks by using learnable task tokens and multi-task co-training, leading to significant performance improvements and robust zero-shot capabilities.
multimodal large language models Task Preference Optimization TPO learnable task tokens multi-task co-training fine-grained visual tasks multimodal capabilities zero-shot capabilities