Jack of All Tasks, Master of Many: Designing General-purpose
  Coarse-to-Fine Vision-Language Model
The ability of large language models (LLMs) to process visual inputs has
given rise to general-purpose vision systems, unifying various vision-language
(VL) tasks by instruction tuning. However, due to the enormous diversity in
input-output formats in the vision domain, existing general-purpose models fail
to successfully integrate segmentation and multi-image inputs with coarse-level
tasks into a single framework. In this work, we introduce VistaLLM, a powerful
visual system that addresses coarse- and fine-grained VL tasks over single and
multiple input images using a unified framework. VistaLLM utilizes an
instruction-guided image tokenizer that filters global embeddings using task
descriptions to extract compressed and refined features from numerous images.
Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to
represent binary segmentation masks as sequences, significantly improving over
previously used uniform sampling. To bolster the desired capability of
VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning
dataset with 6.8M samples. We also address the lack of multi-image grounding
datasets by introducing a novel task, AttCoSeg (Attribute-level
Co-Segmentation), which boosts the model's reasoning and grounding capability
over multiple input images. Extensive experiments on a wide range of V- and VL
tasks demonstrate the effectiveness of VistaLLM by achieving consistent
state-of-the-art performance over strong baselines across all downstream tasks.
Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.
65825b92283fdb0c8de0deca
Shraman Pramanick
admin_assigned
2023-12-20T09:27:22.997Z
64aec10a9135f79b09117476
/avatars/83db978a906782b6df71d178b7a0df89.svg
Shraman Pramanick
user
Shraman
65825b92283fdb0c8de0decb
Guangxing Han
admin_assigned
2023-12-20T09:28:33.248Z
649a2c8fde0fb7f3f4969930
/avatars/3767ad4fa2ffdf414a167b97bbc30352.svg
Guangxing Han
user
GuangxingHan
65825b92283fdb0c8de0decc
Rui Hou
claimed_verified
2023-12-20T08:19:38.770Z
6407d236a7bc7c3865adcc3a
/avatars/8b75cc07fd049acde5363d268c5b88d4.svg
Rui Hou
user
rayhou
65825b92283fdb0c8de0decd
Sayan Nag
admin_assigned
2023-12-20T09:28:40.476Z
61de7e35c201b96c5db3bec6
/avatars/d69c9fcbeff60f6f1c442b2a1226307a.svg
Sayan Nag
user
nagsayan
65825b92283fdb0c8de0dece
Ser-Nam Lim
claimed_verified
2023-12-26T10:37:59.405Z
65884498ce38d143c435d279
/avatars/96a917c40680a79a37dcb8fa75014c21.svg
Ser Nam Lim
user
sernam
65825b92283fdb0c8de0decf
Nicolas Ballas
65825b92283fdb0c8de0ded0
Qifan Wang
admin_assigned
2023-12-20T09:29:33.421Z
64b65c84c991daa61404abd7
/avatars/3592aa8774ea4e2836099e0701b4b78d.svg
WangQifan
user
Bayees
65825b92283fdb0c8de0ded1
Rama Chellappa
65825b92283fdb0c8de0ded2
Amjad Almahairi
admin_assigned
2023-12-20T09:30:13.811Z
63ffacc4fe6383d50b2d2505
/avatars/dd73c355000d0896bebced1c969804f2.svg
Amjad Almahairi
user
aalmah
Jack of All Tasks, Master of Many: Designing General-purpose
  Coarse-to-Fine Vision-Language Model
The ability of large language models (LLMs) to process visual inputs has
given rise to general-purpose vision systems, unifying various vision-language
(VL) tasks by instruction tuning. However, due to the enormous diversity in
input-output formats in the vision domain, existing general-purpose models fail
to successfully integrate segmentation and multi-image inputs with coarse-level
tasks into a single framework. In this work, we introduce VistaLLM, a powerful
visual system that addresses coarse- and fine-grained VL tasks over single and
multiple input images using a unified framework. VistaLLM utilizes an
instruction-guided image tokenizer that filters global embeddings using task
descriptions to extract compressed and refined features from numerous images.
Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to
represent binary segmentation masks as sequences, significantly improving over
previously used uniform sampling. To bolster the desired capability of
VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning
dataset with 6.8M samples. We also address the lack of multi-image grounding
datasets by introducing a novel task, AttCoSeg (Attribute-level
Co-Segmentation), which boosts the model's reasoning and grounding capability
over multiple input images. Extensive experiments on a wide range of V- and VL
tasks demonstrate the effectiveness of VistaLLM by achieving consistent
state-of-the-art performance over strong baselines across all downstream tasks.
Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.
VistaLLM is a unified framework for vision-language tasks that uses instruction-guided tokenization and adaptive sampling to process coarse- and fine-grained visual inputs with state-of-the-art results.
large language models vision-language tasks instruction tuning VistaLLM instruction-guided image tokenizer global embeddings gradient-aware adaptive sampling binary segmentation masks CoinIt multi-image grounding AttCoSeg Attribute-level Co-Segmentation