GiT: Towards Generalist Vision Transformer through Universal Language
  Interface
This paper proposes a simple, yet effective framework, called GiT,
simultaneously applicable for various vision tasks only with a vanilla ViT.
Motivated by the universality of the Multi-layer Transformer architecture (e.g,
GPT) widely used in large language models (LLMs), we seek to broaden its scope
to serve as a powerful vision foundation model (VFM). However, unlike language
modeling, visual tasks typically require specific modules, such as bounding box
heads for detection and pixel decoders for segmentation, greatly hindering the
application of powerful multi-layer transformers in the vision domain. To solve
this, we design a universal language interface that empowers the successful
auto-regressive decoding to adeptly unify various visual tasks, from
image-level understanding (e.g., captioning), over sparse perception (e.g.,
detection), to dense prediction (e.g., segmentation). Based on the above
designs, the entire model is composed solely of a ViT, without any specific
additions, offering a remarkable architectural simplification. GiT is a
multi-task visual model, jointly trained across five representative benchmarks
without task-specific fine-tuning. Interestingly, our GiT builds a new
benchmark in generalist performance, and fosters mutual enhancement across
tasks, leading to significant improvements compared to isolated training. This
reflects a similar impact observed in LLMs. Further enriching training with 27
datasets, GiT achieves strong zero-shot results over various tasks. Due to its
simple design, this paradigm holds promise for narrowing the architectural gap
between vision and language. Code and models will be available at
https://github.com/Haiyang-W/GiT.
65f3b58e8081e7144e0224e6
Haiyang Wang
claimed_verified
2024-03-15T13:15:33.568Z
65f43c3cc9940817caaf4434
/avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
Haiyang Wang
user
Haiyang-W
65f3b58e8081e7144e0224e7
Hao Tang
claimed_verified
2024-03-15T13:15:31.690Z
6585493b53c37507639fe3ba
/avatars/b7e71d4fa5ebb89a7ed6b2a8313687b5.svg
Hao Tang
user
kanashi6
65f3b58e8081e7144e0224e8
Li Jiang
65f3b58e8081e7144e0224e9
Shaoshuai Shi
65f3b58e8081e7144e0224ea
Muhammad Ferjad Naeem
claimed_verified
2025-02-26T15:37:52.342Z
67bf33a512368ec2fad4fe29
/avatars/ea5c03744ec1c2bcc0e6c13efc8f7ddc.svg
Muhammad Ferjad Naeem
user
ferjad
65f3b58e8081e7144e0224eb
Hongsheng Li
admin_assigned
2024-03-15T08:24:42.244Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
65f3b58e8081e7144e0224ec
Bernt Schiele
65f3b58e8081e7144e0224ed
Liwei Wang
admin_assigned
2024-03-15T08:24:59.495Z
6497272b6401266eb6752b89
/avatars/d94a011660f4cad5344756027840dc30.svg
Liwei Wang
user
asdfg80
GiT: Towards Generalist Vision Transformer through Universal Language
  Interface
This paper proposes a simple, yet effective framework, called GiT,
simultaneously applicable for various vision tasks only with a vanilla ViT.
Motivated by the universality of the Multi-layer Transformer architecture (e.g,
GPT) widely used in large language models (LLMs), we seek to broaden its scope
to serve as a powerful vision foundation model (VFM). However, unlike language
modeling, visual tasks typically require specific modules, such as bounding box
heads for detection and pixel decoders for segmentation, greatly hindering the
application of powerful multi-layer transformers in the vision domain. To solve
this, we design a universal language interface that empowers the successful
auto-regressive decoding to adeptly unify various visual tasks, from
image-level understanding (e.g., captioning), over sparse perception (e.g.,
detection), to dense prediction (e.g., segmentation). Based on the above
designs, the entire model is composed solely of a ViT, without any specific
additions, offering a remarkable architectural simplification. GiT is a
multi-task visual model, jointly trained across five representative benchmarks
without task-specific fine-tuning. Interestingly, our GiT builds a new
benchmark in generalist performance, and fosters mutual enhancement across
tasks, leading to significant improvements compared to isolated training. This
reflects a similar impact observed in LLMs. Further enriching training with 27
datasets, GiT achieves strong zero-shot results over various tasks. Due to its
simple design, this paradigm holds promise for narrowing the architectural gap
between vision and language. Code and models will be available at
https://github.com/Haiyang-W/GiT.
GiT is a versatile multi-task vision model using a vanilla ViT architecture that overcomes task-specific limitations and achieves competitive performance across various visual tasks through a universal language interface.
GiT ViT Multi-layer Transformer architecture bounding box heads pixel decoders visual tasks auto-regressive decoding image-level understanding sparse perception dense prediction generalist performance mutual enhancement zero-shot results