CheXStruct and CXReasonBench evaluate Large Vision-Language Models in clinical diagnosis by assessing structured reasoning, visual grounding, and generalization using the MIMIC-CXR-JPG dataset.
Recent progress in Large Vision-Language Models (LVLMs) has enabled promising
applications in medical tasks, such as report generation and visual question
answering. However, existing benchmarks focus mainly on the final diagnostic
answer, offering limited insight into whether models engage in clinically
meaningful reasoning. To address this, we present CheXStruct and CXReasonBench,
a structured pipeline and benchmark built on the publicly available
MIMIC-CXR-JPG dataset. CheXStruct automatically derives a sequence of
intermediate reasoning steps directly from chest X-rays, such as segmenting
anatomical regions, deriving anatomical landmarks and diagnostic measurements,
computing diagnostic indices, and applying clinical thresholds. CXReasonBench
leverages this pipeline to evaluate whether models can perform clinically valid
reasoning steps and to what extent they can learn from structured guidance,
enabling fine-grained and transparent assessment of diagnostic reasoning. The
benchmark comprises 18,988 QA pairs across 12 diagnostic tasks and 1,200 cases,
each paired with up to 4 visual inputs, and supports multi-path, multi-stage
evaluation including visual grounding via anatomical region selection and
diagnostic measurements. Even the strongest of 10 evaluated LVLMs struggle with
structured reasoning and generalization, often failing to link abstract
knowledge with anatomically grounded visual interpretation. The code is
available at https://github.com/ttumyche/CXReasonBench