NL-Eye: Abductive NLI for Images
Will a Visual Language Model (VLM)-based bot warn us about slipping if it
detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
their ability to infer outcomes and causes remains underexplored. To address
this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
abductive reasoning skills. NL-Eye adapts the abductive Natural Language
Inference (NLI) task to the visual domain, requiring models to evaluate the
plausibility of hypothesis images based on a premise image and explain their
decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
images) spanning diverse reasoning categories: physical, functional, logical,
emotional, cultural, and social. The data curation process involved two steps -
writing textual descriptions and generating images using text-to-image models,
both requiring substantial human involvement to ensure high-quality and
challenging scenes. Our experiments show that VLMs struggle significantly on
NL-Eye, often performing at random baseline levels, while humans excel in both
plausibility prediction and explanation quality. This demonstrates a deficiency
in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
crucial step toward developing VLMs capable of robust multimodal reasoning for
real-world applications, including accident-prevention bots and generated video
verification.
67024ddf8572eb2d80f01e3a
Mor Ventura
claimed_verified
2024-10-07T07:26:50.210Z
623f4bd2e801e8c1e59d948e
https://cdn-avatars.huggingface.co/v1/production/uploads/1648315730065-623f4bd2e801e8c1e59d948e.jpeg
Mor Ventura
user
MorVentura
67024ddf8572eb2d80f01e3b
Michael Toker
claimed_verified
2024-10-07T07:26:48.325Z
61c865e3d3702a3bbf50bc04
https://cdn-avatars.huggingface.co/v1/production/uploads/61c865e3d3702a3bbf50bc04/eGgcyeUnOHz0hyap5vhvr.jpeg
Michael Toker
user
tokeron
67024ddf8572eb2d80f01e3c
Nitay Calderon
admin_assigned
2024-10-07T08:44:00.870Z
62d6a0c18faee0ac953c51fa
/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
Nitay Calderon
user
nitay
67024ddf8572eb2d80f01e3d
Zorik Gekhman
67024ddf8572eb2d80f01e3e
Yonatan Bitton
admin_assigned
2024-10-07T08:42:35.195Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
67024ddf8572eb2d80f01e3f
Roi Reichart
NL-Eye: Abductive NLI for Images
Will a Visual Language Model (VLM)-based bot warn us about slipping if it
detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
their ability to infer outcomes and causes remains underexplored. To address
this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
abductive reasoning skills. NL-Eye adapts the abductive Natural Language
Inference (NLI) task to the visual domain, requiring models to evaluate the
plausibility of hypothesis images based on a premise image and explain their
decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
images) spanning diverse reasoning categories: physical, functional, logical,
emotional, cultural, and social. The data curation process involved two steps -
writing textual descriptions and generating images using text-to-image models,
both requiring substantial human involvement to ensure high-quality and
challenging scenes. Our experiments show that VLMs struggle significantly on
NL-Eye, often performing at random baseline levels, while humans excel in both
plausibility prediction and explanation quality. This demonstrates a deficiency
in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
crucial step toward developing VLMs capable of robust multimodal reasoning for
real-world applications, including accident-prevention bots and generated video
verification.
Visual Language Model (VLM) abductive reasoning benchmark abductive Natural Language Inference (NLI) hypothesis images premise image physical functional logical emotional cultural social textual helps text-to-image models plausibility prediction explanation quality multimodal reasoning accident-prevention bots generated video verification