Large Language Models (LLMs) have shown promise for financial applications,
yet their suitability for this high-stakes domain remains largely unproven due
to inadequacies in existing benchmarks. Existing benchmarks solely rely on
score-level evaluation, summarizing performance with a single score that
obscures the nuanced understanding of what models truly know and their precise
limitations. They also rely on datasets that cover only a narrow subset of
financial concepts, while overlooking other essentials for real-world
applications. To address these gaps, we introduce FinCDM, the first cognitive
diagnosis evaluation framework tailored for financial LLMs, enabling the
evaluation of LLMs at the knowledge-skill level, identifying what financial
skills and knowledge they have or lack based on their response patterns across
skill-tagged tasks, rather than a single aggregated number. We construct
CPA-QKA, the first cognitively informed financial evaluation dataset derived
from the Certified Public Accountant (CPA) examination, with comprehensive
coverage of real-world accounting and financial skills. It is rigorously
annotated by domain experts, who author, validate, and annotate questions with
high inter-annotator agreement and fine-grained knowledge labels. Our extensive
experiments on 30 proprietary, open-source, and domain-specific LLMs show that
FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax
and regulatory reasoning overlooked by traditional benchmarks, and uncovers
behavioral clusters among models. FinCDM introduces a new paradigm for
financial LLM evaluation by enabling interpretable, skill-aware diagnosis that
supports more trustworthy and targeted model development, and all datasets and
evaluation scripts will be publicly released to support further research.