Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint
Rebus puzzles, visual riddles that encode language through imagery, spatial
arrangement, and symbolic substitution, pose a unique challenge to current
vision-language models (VLMs). Unlike traditional image captioning or question
answering tasks, rebus solving requires multi-modal abstraction, symbolic
reasoning, and a grasp of cultural, phonetic and linguistic puns. In this
paper, we investigate the capacity of contemporary VLMs to interpret and solve
rebus puzzles by constructing a hand-generated and annotated benchmark of
diverse English-language rebus puzzles, ranging from simple pictographic
substitutions to spatially-dependent cues ("head" over "heels"). We analyze how
different VLMs perform, and our findings reveal that while VLMs exhibit some
surprising capabilities in decoding simple visual clues, they struggle
significantly with tasks requiring abstract reasoning, lateral thinking, and
understanding visual metaphors.
6839d69eb385aa5d7a37bc2b
Heekyung Lee
claimed_verified
2025-06-14T06:24:52.501Z
677c834f3acf9d7d141df32a
/avatars/4b0b46ed4ee914a46df985e69105e0bf.svg
Heekyung Lee
user
Kyunnilee
6839d69eb385aa5d7a37bc2c
Jiaxin Ge
6839d69eb385aa5d7a37bc2d
Tsung-Han Wu
claimed_verified
2025-06-02T07:45:00.174Z
644a767044b75fd95805d232
https://cdn-avatars.huggingface.co/v1/production/uploads/644a767044b75fd95805d232/vHA2vI_B3CpXapdBEwspB.jpeg
Patrick (Tsung-Han) Wu
user
tsunghanwu
6839d69eb385aa5d7a37bc2e
Minwoo Kang
6839d69eb385aa5d7a37bc2f
Trevor Darrell
6839d69eb385aa5d7a37bc30
David M. Chan
Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint
Rebus puzzles, visual riddles that encode language through imagery, spatial
arrangement, and symbolic substitution, pose a unique challenge to current
vision-language models (VLMs). Unlike traditional image captioning or question
answering tasks, rebus solving requires multi-modal abstraction, symbolic
reasoning, and a grasp of cultural, phonetic and linguistic puns. In this
paper, we investigate the capacity of contemporary VLMs to interpret and solve
rebus puzzles by constructing a hand-generated and annotated benchmark of
diverse English-language rebus puzzles, ranging from simple pictographic
substitutions to spatially-dependent cues ("head" over "heels"). We analyze how
different VLMs perform, and our findings reveal that while VLMs exhibit some
surprising capabilities in decoding simple visual clues, they struggle
significantly with tasks requiring abstract reasoning, lateral thinking, and
understanding visual metaphors.
Vision-language models struggle with rebus puzzles, which require abstract reasoning and understanding of visual metaphors, despite performing well on simple visual cues.
vision-language models VLMs multi-modal abstraction symbolic reasoning cultural puns phonetic puns linguistic puns benchmark pictographic substitutions spatially-dependent cues visual metaphors abstract reasoning lateral thinking