HalluSegBench: Counterfactual Visual Reasoning for Segmentation
  Hallucination Evaluation
Recent progress in vision-language segmentation has significantly advanced
grounded visual understanding. However, these models often exhibit
hallucinations by producing segmentation masks for objects not grounded in the
image content or by incorrectly labeling irrelevant regions. Existing
evaluation protocols for segmentation hallucination primarily focus on label or
textual hallucinations without manipulating the visual context, limiting their
capacity to diagnose critical failures. In response, we introduce
HalluSegBench, the first benchmark specifically designed to evaluate
hallucinations in visual grounding through the lens of counterfactual visual
reasoning. Our benchmark consists of a novel dataset of 1340 counterfactual
instance pairs spanning 281 unique object classes, and a set of newly
introduced metrics that quantify hallucination sensitivity under visually
coherent scene edits. Experiments on HalluSegBench with state-of-the-art
vision-language segmentation models reveal that vision-driven hallucinations
are significantly more prevalent than label-driven ones, with models often
persisting in false segmentation, highlighting the need for counterfactual
reasoning to diagnose grounding fidelity.
68681b2c213f123a1f88b96a
Xinzhuo Li
claimed_verified
2025-07-28T08:46:38.959Z
6711d478035ddf7e14b7c537
/avatars/9bf15d8c33c5d544ccb7e3559d029190.svg
Xinzhuo Li
user
LXZ83419
68681b2c213f123a1f88b96b
Adheesh Juvekar
claimed_verified
2025-07-07T11:24:48.443Z
606043321747e63e01e31a09
/avatars/1d38c11a049915a3636169273ce1c8a7.svg
Adheesh Juvekar
user
juvekaradheesh
68681b2c213f123a1f88b96c
Xingyou Liu
68681b2c213f123a1f88b96d
Muntasir Wahed
68681b2c213f123a1f88b96e
Kiet A. Nguyen
68681b2c213f123a1f88b96f
Ismini Lourentzou
claimed_verified
2025-07-05T07:52:43.253Z
678ac3b31cbaa0b4bc295885
/avatars/1244fc1b305c9c6383df9bb5e4707347.svg
Ismini Lourentzou
user
isminoula
HalluSegBench: Counterfactual Visual Reasoning for Segmentation
  Hallucination Evaluation
Recent progress in vision-language segmentation has significantly advanced
grounded visual understanding. However, these models often exhibit
hallucinations by producing segmentation masks for objects not grounded in the
image content or by incorrectly labeling irrelevant regions. Existing
evaluation protocols for segmentation hallucination primarily focus on label or
textual hallucinations without manipulating the visual context, limiting their
capacity to diagnose critical failures. In response, we introduce
HalluSegBench, the first benchmark specifically designed to evaluate
hallucinations in visual grounding through the lens of counterfactual visual
reasoning. Our benchmark consists of a novel dataset of 1340 counterfactual
instance pairs spanning 281 unique object classes, and a set of newly
introduced metrics that quantify hallucination sensitivity under visually
coherent scene edits. Experiments on HalluSegBench with state-of-the-art
vision-language segmentation models reveal that vision-driven hallucinations
are significantly more prevalent than label-driven ones, with models often
persisting in false segmentation, highlighting the need for counterfactual
reasoning to diagnose grounding fidelity.
HalluSegBench evaluates vision-language segmentation models for hallucinations using counterfactual visual reasoning, revealing that vision-driven errors are more common than label-driven ones.
vision-language segmentation grounded visual understanding segmentation masks hallucinations evaluation protocols label hallucinations textual hallucinations counterfactual visual reasoning benchmark dataset metrics hallucination sensitivity scene edits vision-driven hallucinations label-driven hallucinations grounding fidelity