Existing reasoning evaluation frameworks for Large Language Models (LLMs) and
Large Vision-Language Models (LVLMs) predominantly either assess text-based
reasoning or vision-language understanding capabilities, with limited dynamic
interplay between textual and visual constraints. To address this limitation,
we introduce CrossWordBench, a benchmark designed to evaluate the reasoning
capabilities of both LLMs and LVLMs through the medium of crossword puzzles-a
task requiring multimodal adherence to semantic constraints from text-based
clues and intersectional constraints from visual grid structures.
CrossWordBench leverages a controllable puzzle generation framework that
produces puzzles in multiple formats (text and image) and offers different
evaluation strategies ranging from direct puzzle solving to interactive modes.
Our extensive evaluation of over 20 models reveals that reasoning LLMs
outperform non-reasoning models substantially by effectively leveraging
crossing-letter constraints. We further demonstrate that LVLMs struggle with
the task, showing a strong correlation between their puzzle-solving performance
and grid-parsing accuracy. Our findings offer insights into the limitations of
the reasoning capabilities of current LLMs and LVLMs, and provide an effective
approach for creating multimodal constrained tasks for future evaluations.