CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs
  with Controllable Puzzle Generation
Existing reasoning evaluation frameworks for Large Language Models (LLMs) and
Large Vision-Language Models (LVLMs) predominantly either assess text-based
reasoning or vision-language understanding capabilities, with limited dynamic
interplay between textual and visual constraints. To address this limitation,
we introduce CrossWordBench, a benchmark designed to evaluate the reasoning
capabilities of both LLMs and LVLMs through the medium of crossword puzzles-a
task requiring multimodal adherence to semantic constraints from text-based
clues and intersectional constraints from visual grid structures.
CrossWordBench leverages a controllable puzzle generation framework that
produces puzzles in multiple formats (text and image) and offers different
evaluation strategies ranging from direct puzzle solving to interactive modes.
Our extensive evaluation of over 20 models reveals that reasoning LLMs
outperform non-reasoning models substantially by effectively leveraging
crossing-letter constraints. We further demonstrate that LVLMs struggle with
the task, showing a strong correlation between their puzzle-solving performance
and grid-parsing accuracy. Our findings offer insights into the limitations of
the reasoning capabilities of current LLMs and LVLMs, and provide an effective
approach for creating multimodal constrained tasks for future evaluations.
67ec9d4ad327ed17ec707488
Jixuan Leng
admin_assigned
2025-04-09T16:47:14.691Z
64811214cacb1c4a06988bd9
/avatars/538a3029878c2200df049b54c97c9327.svg
Jixuan Leng
user
JixuanLeng
67ec9d4ad327ed17ec707489
Chengsong Huang
admin_assigned
2025-04-09T16:47:27.410Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
user
ChengsongHuang
67ec9d4ad327ed17ec70748a
Langlin Huang
67ec9d4ad327ed17ec70748b
Bill Yuchen Lin
admin_assigned
2025-04-09T16:47:44.967Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
67ec9d4ad327ed17ec70748c
William W. Cohen
admin_assigned
2025-04-09T16:48:09.463Z
66a28706d3449709d6943ef2
/avatars/804f45be500d40e1c0e973c691dd2c73.svg
William Cohen
user
wwcohen
67ec9d4ad327ed17ec70748d
Haohan Wang
admin_assigned
2025-04-09T16:48:23.506Z
670d920206218f52e8ea376d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/j_-zv1ftLCJbjjw9-wnN8.png
Haohan Wang
user
haohanw
67ec9d4ad327ed17ec70748e
Jiaxin Huang
CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs
  with Controllable Puzzle Generation
Existing reasoning evaluation frameworks for Large Language Models (LLMs) and
Large Vision-Language Models (LVLMs) predominantly either assess text-based
reasoning or vision-language understanding capabilities, with limited dynamic
interplay between textual and visual constraints. To address this limitation,
we introduce CrossWordBench, a benchmark designed to evaluate the reasoning
capabilities of both LLMs and LVLMs through the medium of crossword puzzles-a
task requiring multimodal adherence to semantic constraints from text-based
clues and intersectional constraints from visual grid structures.
CrossWordBench leverages a controllable puzzle generation framework that
produces puzzles in multiple formats (text and image) and offers different
evaluation strategies ranging from direct puzzle solving to interactive modes.
Our extensive evaluation of over 20 models reveals that reasoning LLMs
outperform non-reasoning models substantially by effectively leveraging
crossing-letter constraints. We further demonstrate that LVLMs struggle with
the task, showing a strong correlation between their puzzle-solving performance
and grid-parsing accuracy. Our findings offer insights into the limitations of
the reasoning capabilities of current LLMs and LVLMs, and provide an effective
approach for creating multimodal constrained tasks for future evaluations.
CrossWordBench evaluates reasoning capabilities of LLMs and LVLMs using crossword puzzles, highlighting differences in textual and visual constraint handling.
CrossWordBench Large Language Models Large Vision-Language Models multimodal adherence semantic constraints visual grid structures controllable puzzle generation crossing-letter constraints grid-parsing accuracy