SEAM: Semantically Equivalent Across Modalities Benchmark for
  Vision-Language Models
Evaluating whether vision-language models (VLMs) reason consistently across
representations is challenging because modality comparisons are typically
confounded by task differences and asymmetric information. We introduce SEAM, a
benchmark that pairs semantically equivalent inputs across four domains that
have existing standardized textual and visual notations. By employing distinct
notation systems across modalities, in contrast to OCR-based image-text
pairing, SEAM provides a rigorous comparative assessment of the
textual-symbolic and visual-spatial reasoning capabilities of VLMs. Across 21
contemporary models, we observe systematic modality imbalance: vision
frequently lags language in overall performance, despite the problems
containing semantically equivalent information, and cross-modal agreement is
relatively low. Our error analysis reveals two main drivers: textual perception
failures from tokenization in domain notation and visual perception failures
that induce hallucinations. We also show that our results are largely robust to
visual transformations. SEAM establishes a controlled, semantically equivalent
setting for measuring and improving modality-agnostic reasoning.
68af95f3245176306494cb5a
Zhenwei Tang
claimed_verified
2025-08-28T08:55:17.678Z
653964f156c9b35961f44456
/avatars/0245e7543bff797405a927df5d16aeca.svg
Joseph Tang
user
lilvjosephtang
68af95f3245176306494cb5b
Difan Jiao
68af95f3245176306494cb5c
Blair Yang
68af95f3245176306494cb5d
Ashton Anderson
SEAM: Semantically Equivalent Across Modalities Benchmark for
  Vision-Language Models
Evaluating whether vision-language models (VLMs) reason consistently across
representations is challenging because modality comparisons are typically
confounded by task differences and asymmetric information. We introduce SEAM, a
benchmark that pairs semantically equivalent inputs across four domains that
have existing standardized textual and visual notations. By employing distinct
notation systems across modalities, in contrast to OCR-based image-text
pairing, SEAM provides a rigorous comparative assessment of the
textual-symbolic and visual-spatial reasoning capabilities of VLMs. Across 21
contemporary models, we observe systematic modality imbalance: vision
frequently lags language in overall performance, despite the problems
containing semantically equivalent information, and cross-modal agreement is
relatively low. Our error analysis reveals two main drivers: textual perception
failures from tokenization in domain notation and visual perception failures
that induce hallucinations. We also show that our results are largely robust to
visual transformations. SEAM establishes a controlled, semantically equivalent
setting for measuring and improving modality-agnostic reasoning.
SEAM is a benchmark that evaluates vision-language models' reasoning consistency across modalities using semantically equivalent inputs, revealing systematic modality imbalance and visual hallucinations.
vision-language models VLMs benchmark semantically equivalent inputs textual-symbolic reasoning visual-spatial reasoning OCR-based image-text pairing tokenization visual perception failures visual hallucinations visual transformations modality-agnostic reasoning