Measuring Progress in Fine-grained Vision-and-Language Understanding
While pretraining on large-scale image-text data from the Web has facilitated
rapid progress on many vision-and-language (V&L) tasks, recent work has
demonstrated that pretrained models lack "fine-grained" understanding, such as
the ability to recognise relationships, verbs, and numbers in images. This has
resulted in an increased interest in the community to either develop new
benchmarks or models for such capabilities. To better understand and quantify
progress in this direction, we investigate four competitive V&L models on four
fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al.,
2022) consistently outperforms other baselines, and that modelling innovations
can impact performance more than scaling Web data, which even degrades
performance sometimes. Through a deeper investigation of X-VLM, we highlight
the importance of both novel losses and rich data sources for learning
fine-grained skills. Finally, we inspect training dynamics, and discover that
for some tasks, performance peaks early in training or significantly
fluctuates, never converging.
6461cb755ea1bc9a9da94cda
Emanuele Bugliarello
claimed_verified
2023-05-15T15:28:30.074Z
626a9284783d5891f45beb53
/avatars/8737662babc5d42cafba6087ab33e716.svg
Emanuele Bugliarello
user
e-bug
6461cb755ea1bc9a9da94cdb
Laurent Sartran
6461cb755ea1bc9a9da94cdc
Aishwarya Agrawal
admin_assigned
2023-05-26T12:12:50.228Z
644fff2d17b6189cda58cb0e
/avatars/03b367b9033c655129e48bd94bd245ee.svg
Aishwarya Agrawal
user
aagrawal
6461cb755ea1bc9a9da94cdd
Lisa Anne Hendricks
6461cb755ea1bc9a9da94cde
Aida Nematzadeh
Measuring Progress in Fine-grained Vision-and-Language Understanding
While pretraining on large-scale image-text data from the Web has facilitated
rapid progress on many vision-and-language (V&L) tasks, recent work has
demonstrated that pretrained models lack "fine-grained" understanding, such as
the ability to recognise relationships, verbs, and numbers in images. This has
resulted in an increased interest in the community to either develop new
benchmarks or models for such capabilities. To better understand and quantify
progress in this direction, we investigate four competitive V&L models on four
fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al.,
2022) consistently outperforms other baselines, and that modelling innovations
can impact performance more than scaling Web data, which even degrades
performance sometimes. Through a deeper investigation of X-VLM, we highlight
the importance of both novel losses and rich data sources for learning
fine-grained skills. Finally, we inspect training dynamics, and discover that
for some tasks, performance peaks early in training or significantly
fluctuates, never converging.
Analysis of competitive vision-and-language models on fine-grained benchmarks reveals that X-VLM outperforms others, emphasizing the impact of innovative losses and rich data sources more than increased Web data scaling.
vision-and-language models fine-grained understanding X-VLM novel losses rich data sources training dynamics