VisIT-Bench: A Benchmark for Vision-Language Instruction Following
  Inspired by Real-World Use
We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for
evaluation of instruction-following vision-language models for real-world use.
Our starting point is curating 70 'instruction families' that we envision
instruction tuned vision-language models should be able to address. Extending
beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to
game playing and creative generation. Following curation, our dataset comprises
592 test queries, each with a human-authored instruction-conditioned caption.
These descriptions surface instruction-specific factors, e.g., for an
instruction asking about the accessibility of a storefront for wheelchair
users, the instruction-conditioned caption describes ramps/potential obstacles.
These descriptions enable 1) collecting human-verified reference outputs for
each instance; and 2) automatic evaluation of candidate multimodal generations
using a text-only LLM, aligning with human judgment. We quantify quality gaps
between models and references using both human and automatic evaluations; e.g.,
the top-performing instruction-following model wins against the GPT-4 reference
in just 27% of the comparison. VisIT-Bench is dynamic to participate,
practitioners simply submit their model's response on the project website;
Data, code and leaderboard is available at visit-bench.github.io.
64db0bfc00b80a024c5924f7
Yonatan Bitton
extracted_confirmed
2023-08-15T12:21:14.755Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
64db0bfc00b80a024c5924f8
Hritik Bansal
claimed_verified
2023-08-15T12:56:40.216Z
61c5c25705aa54027c52f7b3
/avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
Hritik Bansal
user
hbXNov
64db0bfc00b80a024c5924f9
Jack Hessel
admin_assigned
2023-08-15T13:26:23.279Z
625498644f4edf771516b2cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
Jack Hessel
user
jmhessel
64db0bfc00b80a024c5924fa
Rulin Shao
admin_assigned
2023-08-15T13:26:42.433Z
6334a0bd31a2be3938c59537
https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
Rulin Shao
user
rulins
64db0bfc00b80a024c5924fb
Wanrong Zhu
claimed_verified
2023-08-16T13:23:22.828Z
633a7e91c8eb7b089034fb3c
/avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
Wanrong Zhu
user
VegB
64db0bfc00b80a024c5924fc
Anas Awadalla
admin_assigned
2023-08-15T13:27:09.283Z
60f1ab3afe0d78e01037eeb1
/avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
Anas Awadalla
user
anas-awadalla
64db0bfc00b80a024c5924fd
Josh Gardner
admin_assigned
2023-08-15T13:27:47.014Z
63e6d6b22d2c508de90153a1
/avatars/11d653e5376bd122ff82778a8393f541.svg
Josh Gardner
user
jpgard
64db0bfc00b80a024c5924fe
Rohan Taori
admin_assigned
2023-08-15T13:27:56.760Z
622fca0d8d46dde5d605f83e
https://cdn-avatars.huggingface.co/v1/production/uploads/1647489114181-622fca0d8d46dde5d605f83e.jpeg
Rohan Taori
user
rtaori
64db0bfc00b80a024c5924ff
Ludwig Schimdt
VisIT-Bench: A Benchmark for Vision-Language Instruction Following
  Inspired by Real-World Use
We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for
evaluation of instruction-following vision-language models for real-world use.
Our starting point is curating 70 'instruction families' that we envision
instruction tuned vision-language models should be able to address. Extending
beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to
game playing and creative generation. Following curation, our dataset comprises
592 test queries, each with a human-authored instruction-conditioned caption.
These descriptions surface instruction-specific factors, e.g., for an
instruction asking about the accessibility of a storefront for wheelchair
users, the instruction-conditioned caption describes ramps/potential obstacles.
These descriptions enable 1) collecting human-verified reference outputs for
each instance; and 2) automatic evaluation of candidate multimodal generations
using a text-only LLM, aligning with human judgment. We quantify quality gaps
between models and references using both human and automatic evaluations; e.g.,
the top-performing instruction-following model wins against the GPT-4 reference
in just 27% of the comparison. VisIT-Bench is dynamic to participate,
practitioners simply submit their model's response on the project website;
Data, code and leaderboard is available at visit-bench.github.io.
VisIT-Bench evaluates instruction-following vision-language models across a variety of tasks through human-authored instructions and multimodal generation assessments.
instruction-following vision-language models instruction families VQAv2 COCO instruction-conditioned caption multimodal generation automatic evaluation text-only LLM leaderboard