VCR: Visual Caption Restoration
We introduce Visual Caption Restoration (VCR), a novel vision-language task
that challenges models to accurately restore partially obscured texts using
pixel-level hints within images. This task stems from the observation that text
embedded in images is intrinsically different from common visual elements and
natural language due to the need to align the modalities of vision, text, and
text embedded in images. While numerous works have integrated text embedded in
images into visual question-answering tasks, approaches to these tasks
generally rely on optical character recognition or masked language modeling,
thus reducing the task to mainly text-based processing. However, text-based
processing becomes ineffective in VCR as accurate text restoration depends on
the combined information from provided images, context, and subtle cues from
the tiny exposed areas of masked texts. We develop a pipeline to generate
synthetic images for the VCR task using image-caption pairs, with adjustable
caption visibility to control the task difficulty. With this pipeline, we
construct a dataset for VCR called VCR-Wiki using images with captions from
Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
hard split variants. Our results reveal that current vision language models
significantly lag behind human performance in the VCR task, and merely
fine-tuning the models on our dataset does not lead to notable improvements. We
release VCR-Wiki and the data construction code to facilitate future research.
6667b702dd968ec963cb9dee
Tianyu Zhang
claimed_verified
2024-06-13T18:36:20.440Z
6452d79149b6b9a2383b5775
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T28lP0kE7PZIGzJjhSpSx.jpeg
Tianyu Zhang
user
TianyuZhang
6667b702dd968ec963cb9def
Suyuchen Wang
claimed_verified
2024-06-11T06:59:10.569Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/MWNanci3x5g780xh-704U.png
Suyuchen Wang
user
sheryc
6667b702dd968ec963cb9df0
Lu Li
claimed_verified
2024-06-15T21:29:05.165Z
658fbec068d0b76331a81e38
https://cdn-avatars.huggingface.co/v1/production/uploads/658fbec068d0b76331a81e38/E-1fyTCM4eltkWOXwWU5k.jpeg
Lu Li
user
luli2949
6667b702dd968ec963cb9df1
Ge Zhang
claimed_verified
2024-06-21T07:07:15.807Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6667b702dd968ec963cb9df2
Perouz Taslakian
6667b702dd968ec963cb9df3
Sai Rajeswar
claimed_verified
2024-06-27T09:47:30.661Z
642f99079b2484d7d857341b
/avatars/01965cc5a5dbe9c08025a51973462a6a.svg
Sai Rajeswar
user
rajeswarsai
6667b702dd968ec963cb9df4
Jie Fu
claimed_verified
2024-06-24T07:33:20.052Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
6667b702dd968ec963cb9df5
Bang Liu
claimed_verified
2024-06-14T07:24:35.896Z
654a97282d2fcd6bf2851173
https://cdn-avatars.huggingface.co/v1/production/uploads/654a97282d2fcd6bf2851173/9zXf940gr4WNt4e-oOt4k.png
Bang Liu
user
Bang-UdeM-Mila
6667b702dd968ec963cb9df6
Yoshua Bengio
VCR: Visual Caption Restoration
We introduce Visual Caption Restoration (VCR), a novel vision-language task
that challenges models to accurately restore partially obscured texts using
pixel-level hints within images. This task stems from the observation that text
embedded in images is intrinsically different from common visual elements and
natural language due to the need to align the modalities of vision, text, and
text embedded in images. While numerous works have integrated text embedded in
images into visual question-answering tasks, approaches to these tasks
generally rely on optical character recognition or masked language modeling,
thus reducing the task to mainly text-based processing. However, text-based
processing becomes ineffective in VCR as accurate text restoration depends on
the combined information from provided images, context, and subtle cues from
the tiny exposed areas of masked texts. We develop a pipeline to generate
synthetic images for the VCR task using image-caption pairs, with adjustable
caption visibility to control the task difficulty. With this pipeline, we
construct a dataset for VCR called VCR-Wiki using images with captions from
Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
hard split variants. Our results reveal that current vision language models
significantly lag behind human performance in the VCR task, and merely
fine-tuning the models on our dataset does not lead to notable improvements. We
release VCR-Wiki and the data construction code to facilitate future research.
A new vision-language task, Visual Caption Restoration, challenges models to restore partially obscured texts using pixel-level hints, revealing that current models underperform compared to human performance.
vision-language task Visual Caption Restoration pixel-level hints optical character recognition masked language modeling synthetic images image-caption pairs VCR-Wiki vision language models