VisionLLM: Large Language Model is also an Open-Ended Decoder for
  Vision-Centric Tasks
Large language models (LLMs) have notably accelerated progress towards
artificial general intelligence (AGI), with their impressive zero-shot capacity
for user-tailored tasks, endowing them with immense potential across a range of
applications. However, in the field of computer vision, despite the
availability of numerous powerful vision foundation models (VFMs), they are
still restricted to tasks in a pre-defined form, struggling to match the
open-ended task capabilities of LLMs. In this work, we present an LLM-based
framework for vision-centric tasks, termed VisionLLM. This framework provides a
unified perspective for vision and language tasks by treating images as a
foreign language and aligning vision-centric tasks with language tasks that can
be flexibly defined and managed using language instructions. An LLM-based
decoder can then make appropriate predictions based on these instructions for
open-ended tasks. Extensive experiments show that the proposed VisionLLM can
achieve different levels of task customization through language instructions,
from fine-grained object-level to coarse-grained task-level customization, all
with good results. It's noteworthy that, with a generalist LLM-based framework,
our model can achieve over 60\% mAP on COCO, on par with detection-specific
models. We hope this model can set a new baseline for generalist vision and
language models. The demo shall be released based on
https://github.com/OpenGVLab/InternGPT. The code shall be released at
https://github.com/OpenGVLab/VisionLLM.
6466fcce14e059dde8bc6356
Wenhai Wang
6466fcce14e059dde8bc6357
Zhe Chen
claimed_verified
2023-05-19T11:11:56.893Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
6466fcce14e059dde8bc6358
Xiaokang Chen
claimed_verified
2025-02-05T15:57:22.272Z
6635e701420baf7bc3f93561
/avatars/cdbb3085fee73ac520888977e2c575ea.svg
Xiaokang Chen
user
CharlesCXK
6466fcce14e059dde8bc6359
Jiannan Wu
claimed_verified
2025-08-08T16:22:37.745Z
64675a52efbd7ae30970c911
/avatars/149c6d813956e228567811770df01859.svg
Jiannan Wu
user
wjn922
6466fcce14e059dde8bc635a
Xizhou Zhu
6466fcce14e059dde8bc635b
Gang Zeng
6466fcce14e059dde8bc635c
Ping Luo
6466fcce14e059dde8bc635d
Tong Lu
6466fcce14e059dde8bc635e
Jie Zhou
6466fcce14e059dde8bc635f
Yu Qiao
6466fcce14e059dde8bc6360
Jifeng Dai
claimed_verified
2023-05-22T09:36:54.865Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
VisionLLM: Large Language Model is also an Open-Ended Decoder for
  Vision-Centric Tasks
Large language models (LLMs) have notably accelerated progress towards
artificial general intelligence (AGI), with their impressive zero-shot capacity
for user-tailored tasks, endowing them with immense potential across a range of
applications. However, in the field of computer vision, despite the
availability of numerous powerful vision foundation models (VFMs), they are
still restricted to tasks in a pre-defined form, struggling to match the
open-ended task capabilities of LLMs. In this work, we present an LLM-based
framework for vision-centric tasks, termed VisionLLM. This framework provides a
unified perspective for vision and language tasks by treating images as a
foreign language and aligning vision-centric tasks with language tasks that can
be flexibly defined and managed using language instructions. An LLM-based
decoder can then make appropriate predictions based on these instructions for
open-ended tasks. Extensive experiments show that the proposed VisionLLM can
achieve different levels of task customization through language instructions,
from fine-grained object-level to coarse-grained task-level customization, all
with good results. It's noteworthy that, with a generalist LLM-based framework,
our model can achieve over 60\% mAP on COCO, on par with detection-specific
models. We hope this model can set a new baseline for generalist vision and
language models. The demo shall be released based on
https://github.com/OpenGVLab/InternGPT. The code shall be released at
https://github.com/OpenGVLab/VisionLLM.
A unified LLM-based framework, VisionLLM, treats images as a foreign language, enabling open-ended vision tasks with language instructions and achieves performance comparable to detection-specific models.
large language models artificial general intelligence vision foundation models foreign language language instructions mAP COCO generalist vision and language models