NL-Eye: Abductive NLI for Images
Will a Visual Language Model (VLM)-based bot warn us about slipping if it
detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
their ability to infer outcomes and causes remains underexplored. To address
this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
abductive reasoning skills. NL-Eye adapts the abductive Natural Language
Inference (NLI) task to the visual domain, requiring models to evaluate the
plausibility of hypothesis images based on a premise image and explain their
decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
images) spanning diverse reasoning categories: physical, functional, logical,
emotional, cultural, and social. The data curation process involved two steps -
writing textual descriptions and generating images using text-to-image models,
both requiring substantial human involvement to ensure high-quality and
challenging scenes. Our experiments show that VLMs struggle significantly on
NL-Eye, often performing at random baseline levels, while humans excel in both
plausibility prediction and explanation quality. This demonstrates a deficiency
in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
crucial step toward developing VLMs capable of robust multimodal reasoning for
real-world applications, including accident-prevention bots and generated video
verification.
67024ddf8572eb2d80f01e3a
Mor Ventura
claimed_verified
2024-10-07T07:26:50.210Z
623f4bd2e801e8c1e59d948e
https://cdn-avatars.huggingface.co/v1/production/uploads/1648315730065-623f4bd2e801e8c1e59d948e.jpeg
Mor Ventura
user
MorVentura
67024ddf8572eb2d80f01e3b
Michael Toker
claimed_verified
2024-10-07T07:26:48.325Z
61c865e3d3702a3bbf50bc04
https://cdn-avatars.huggingface.co/v1/production/uploads/61c865e3d3702a3bbf50bc04/eGgcyeUnOHz0hyap5vhvr.jpeg
Michael Toker
user
tokeron
67024ddf8572eb2d80f01e3c
Nitay Calderon
admin_assigned
2024-10-07T08:44:00.870Z
62d6a0c18faee0ac953c51fa
/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
Nitay Calderon
user
nitay
67024ddf8572eb2d80f01e3d
Zorik Gekhman
67024ddf8572eb2d80f01e3e
Yonatan Bitton
admin_assigned
2024-10-07T08:42:35.195Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
67024ddf8572eb2d80f01e3f
Roi Reichart
NL-Eye: Abductive NLI for Images
Will a Visual Language Model (VLM)-based bot warn us about slipping if it
detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
their ability to infer outcomes and causes remains underexplored. To address
this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
abductive reasoning skills. NL-Eye adapts the abductive Natural Language
Inference (NLI) task to the visual domain, requiring models to evaluate the
plausibility of hypothesis images based on a premise image and explain their
decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
images) spanning diverse reasoning categories: physical, functional, logical,
emotional, cultural, and social. The data curation process involved two steps -
writing textual descriptions and generating images using text-to-image models,
both requiring substantial human involvement to ensure high-quality and
challenging scenes. Our experiments show that VLMs struggle significantly on
NL-Eye, often performing at random baseline levels, while humans excel in both
plausibility prediction and explanation quality. This demonstrates a deficiency
in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
crucial step toward developing VLMs capable of robust multimodal reasoning for
real-world applications, including accident-prevention bots and generated video
verification.
NL-Eye is a benchmark that assesses VLMs' visual abductive reasoning skills by evaluating their ability to predict and explain the plausibility of hypothesis images based on premise images.
Visual Language Model VLM NL-Eye abductive reasoning abductive Natural Language Inference NLI text-to-image models multimodal reasoning accident-prevention bots generated video verification