Harnessing Webpage UIs for Text-Rich Visual Understanding
Text-rich visual understanding-the ability to process environments where
dense textual content is integrated with visuals-is crucial for multimodal
large language models (MLLMs) to interact effectively with structured
environments. To enhance this capability, we propose synthesizing general
multimodal instructions from webpage UIs using text-based large language models
(LLMs). Despite lacking direct visual input, text-based LLMs are able to
process structured text representations from webpage accessibility trees. These
instructions are then paired with UI screenshots to train multimodal models. We
introduce MultiUI, a dataset containing 7.3 million samples from 1 million
websites, covering diverse multimodal tasks and UI layouts. Models trained on
MultiUI not only excel in web UI tasks-achieving up to a 48\% improvement on
VisualWebBench and a 19.1\% boost in action accuracy on a web agent dataset
Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to
non-UI domains, such as document understanding, OCR, and chart interpretation.
These results highlight the broad applicability of web UI data for advancing
text-rich visual understanding across various scenarios.
6711c43e0babf0ff2ec5f194
Junpeng Liu
claimed_verified
2024-10-18T08:12:38.288Z
6512a2e284bedffb5ac5a511
/avatars/e798f2c8633d77aea41fce684d749390.svg
Junpeng Liu
user
jeepliu
6711c43e0babf0ff2ec5f195
Tianyue Ou
admin_assigned
2024-10-18T09:57:06.209Z
65403d8781a8731a1c09a584
https://cdn-avatars.huggingface.co/v1/production/uploads/65403d8781a8731a1c09a584/858OEZnMxbJleFiI1UgGs.jpeg
Tianyue Ou
user
oottyy
6711c43e0babf0ff2ec5f196
Yifan Song
6711c43e0babf0ff2ec5f197
Yuxiao Qu
admin_assigned
2024-10-18T12:33:26.181Z
6500bbf5e102da55f9ed43fc
https://cdn-avatars.huggingface.co/v1/production/uploads/6500bbf5e102da55f9ed43fc/QZ6EAFV2CStFsILmTJw5D.jpeg
Yuxiao Qu
user
CohenQu
6711c43e0babf0ff2ec5f198
Wai Lam
6711c43e0babf0ff2ec5f199
Chenyan Xiong
admin_assigned
2024-10-18T12:33:36.903Z
617ae49bc0ff6006217aa22e
https://cdn-avatars.huggingface.co/v1/production/uploads/1635543252231-617ae49bc0ff6006217aa22e.jpeg
Chenyan Xiong
user
xiongchenyan
6711c43e0babf0ff2ec5f19a
Wenhu Chen
admin_assigned
2024-10-18T12:33:42.084Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
6711c43e0babf0ff2ec5f19b
Graham Neubig
admin_assigned
2024-10-18T12:33:49.506Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
6711c43e0babf0ff2ec5f19c
Xiang Yue
claimed_verified
2024-10-18T08:12:35.934Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
Harnessing Webpage UIs for Text-Rich Visual Understanding
Text-rich visual understanding-the ability to process environments where
dense textual content is integrated with visuals-is crucial for multimodal
large language models (MLLMs) to interact effectively with structured
environments. To enhance this capability, we propose synthesizing general
multimodal instructions from webpage UIs using text-based large language models
(LLMs). Despite lacking direct visual input, text-based LLMs are able to
process structured text representations from webpage accessibility trees. These
instructions are then paired with UI screenshots to train multimodal models. We
introduce MultiUI, a dataset containing 7.3 million samples from 1 million
websites, covering diverse multimodal tasks and UI layouts. Models trained on
MultiUI not only excel in web UI tasks-achieving up to a 48\% improvement on
VisualWebBench and a 19.1\% boost in action accuracy on a web agent dataset
Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to
non-UI domains, such as document understanding, OCR, and chart interpretation.
These results highlight the broad applicability of web UI data for advancing
text-rich visual understanding across various scenarios.
MultiUI, a dataset from webpage UIs, enhances multimodal models' ability to handle text-rich visual environments, improving performance on web and non-web tasks.
multimodal large language models MLLMs text-based large language models LLMs accessibility trees multimodal instructions UI screenshots VisualWebBench Mind2Web action accuracy document understanding OCR chart interpretation