Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models
Modern Large Vision-Language Models (LVLMs) enjoy the same vision vocabulary
-- CLIP, which can cover most common vision tasks. However, for some special
vision task that needs dense and fine-grained vision perception, e.g.,
document-level OCR or chart understanding, especially in non-English scenarios,
the CLIP-style vocabulary may encounter low efficiency in tokenizing the vision
knowledge and even suffer out-of-vocabulary problem. Accordingly, we propose
Vary, an efficient and effective method to scale up the vision vocabulary of
LVLMs. The procedures of Vary are naturally divided into two folds: the
generation and integration of a new vision vocabulary. In the first phase, we
devise a vocabulary network along with a tiny decoder-only transformer to
produce the desired vocabulary via autoregression. In the next, we scale up the
vanilla vision vocabulary by merging the new one with the original one (CLIP),
enabling the LVLMs can quickly garner new features. Compared to the popular
BLIP-2, MiniGPT4, and LLaVA, Vary can maintain its vanilla capabilities while
enjoying more excellent fine-grained perception and understanding ability.
Specifically, Vary is competent in new document parsing features (OCR or
markdown conversion) while achieving 78.2% ANLS in DocVQA and 36.2% in MMVet.
Our code will be publicly available on the homepage.
6577dcc49fae206bdf9fe91d
Haoran Wei
admin_assigned
2023-12-12T08:09:47.086Z
6436618aeef1f55654a9f458
https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
Haoran Wei
user
HaoranWei
6577dcc49fae206bdf9fe91e
Lingyu Kong
claimed_verified
2024-04-18T08:11:22.548Z
650d23b69169ea733178f508
/avatars/5f9a7ab6b6a44e74c3eb6a59da2e21ed.svg
Lingyu Kong
user
kppkkp
6577dcc49fae206bdf9fe91f
Jinyue Chen
6577dcc49fae206bdf9fe920
Liang Zhao
6577dcc49fae206bdf9fe921
Zheng Ge
6577dcc49fae206bdf9fe922
Jinrong Yang
admin_assigned
2023-12-12T08:11:36.650Z
650bd10c1f2949b99e65ef23
/avatars/8f36f844690b64bcc7c358063f2bdc3b.svg
Jinrong Yang
user
YancieYJR
6577dcc49fae206bdf9fe923
Jianjian Sun
6577dcc49fae206bdf9fe924
Chunrui Han
6577dcc49fae206bdf9fe925
Xiangyu Zhang
Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models
Modern Large Vision-Language Models (LVLMs) enjoy the same vision vocabulary
-- CLIP, which can cover most common vision tasks. However, for some special
vision task that needs dense and fine-grained vision perception, e.g.,
document-level OCR or chart understanding, especially in non-English scenarios,
the CLIP-style vocabulary may encounter low efficiency in tokenizing the vision
knowledge and even suffer out-of-vocabulary problem. Accordingly, we propose
Vary, an efficient and effective method to scale up the vision vocabulary of
LVLMs. The procedures of Vary are naturally divided into two folds: the
generation and integration of a new vision vocabulary. In the first phase, we
devise a vocabulary network along with a tiny decoder-only transformer to
produce the desired vocabulary via autoregression. In the next, we scale up the
vanilla vision vocabulary by merging the new one with the original one (CLIP),
enabling the LVLMs can quickly garner new features. Compared to the popular
BLIP-2, MiniGPT4, and LLaVA, Vary can maintain its vanilla capabilities while
enjoying more excellent fine-grained perception and understanding ability.
Specifically, Vary is competent in new document parsing features (OCR or
markdown conversion) while achieving 78.2% ANLS in DocVQA and 36.2% in MMVet.
Our code will be publicly available on the homepage.
Vary enhances Vision-Language Models with a new scalable and fine-grained vision vocabulary through autoregressive generation and integration, improving performance in tasks like document OCR and chart understanding.
Large Vision-Language Models LVLMs CLIP document-level OCR chart understanding vision vocabulary vocabulary network tiny decoder-only transformer autoregression BLIP-2 MiniGPT4 LLaVA DocVQA MMVet