Enhancing Vision-Language Pre-training with Rich Supervisions
We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
pre-training paradigm for Vision-Language Models using data from large-scale
web screenshot rendering. Using web screenshots unlocks a treasure trove of
visual and textual cues that are not present in using image-text pairs. In S4,
we leverage the inherent tree-structured hierarchy of HTML elements and the
spatial localization to carefully design 10 pre-training tasks with large scale
annotated data. These tasks resemble downstream tasks across different domains
and the annotations are cheap to obtain. We demonstrate that, compared to
current screenshot pre-training objectives, our innovative pre-training method
significantly enhances performance of image-to-text model in nine varied and
popular downstream tasks - up to 76.1% improvements on Table Detection, and at
least 1% on Widget Captioning.
65e93d92c331a092145618f8
Yuan Gao
65e93d92c331a092145618f9
Kunyu Shi
claimed_verified
2024-03-07T16:46:06.124Z
65e9ea877c911508683a27a2
/avatars/eb0a6155a6bdce7d62d9b04cc9fedafa.svg
Kunyu Shi
user
shikunyu8
65e93d92c331a092145618fa
Pengkai Zhu
65e93d92c331a092145618fb
Edouard Belval
65e93d92c331a092145618fc
Oren Nuriel
65e93d92c331a092145618fd
Srikar Appalaraju
65e93d92c331a092145618fe
Shabnam Ghadar
65e93d92c331a092145618ff
Vijay Mahadevan
65e93d92c331a09214561900
Zhuowen Tu
65e93d92c331a09214561901
Stefano Soatto
Enhancing Vision-Language Pre-training with Rich Supervisions
We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
pre-training paradigm for Vision-Language Models using data from large-scale
web screenshot rendering. Using web screenshots unlocks a treasure trove of
visual and textual cues that are not present in using image-text pairs. In S4,
we leverage the inherent tree-structured hierarchy of HTML elements and the
spatial localization to carefully design 10 pre-training tasks with large scale
annotated data. These tasks resemble downstream tasks across different domains
and the annotations are cheap to obtain. We demonstrate that, compared to
current screenshot pre-training objectives, our innovative pre-training method
significantly enhances performance of image-to-text model in nine varied and
popular downstream tasks - up to 76.1% improvements on Table Detection, and at
least 1% on Widget Captioning.
Strongly Supervised pre-training with ScreenShots (S4) enhances Vision-Language Models by using web screenshots with tree-structured HTML elements and spatial localization, resulting in significant improvements in diverse downstream tasks.
Strongly Supervised pre-training ScreenShots Vision-Language Models web screenshots tree-structured hierarchy HTML elements spatial localization pre-training tasks image-to-text model Table Detection Widget Captioning