From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating
  Financial Large Language Models
Large Language Models (LLMs) have shown promise for financial applications,
yet their suitability for this high-stakes domain remains largely unproven due
to inadequacies in existing benchmarks. Existing benchmarks solely rely on
score-level evaluation, summarizing performance with a single score that
obscures the nuanced understanding of what models truly know and their precise
limitations. They also rely on datasets that cover only a narrow subset of
financial concepts, while overlooking other essentials for real-world
applications. To address these gaps, we introduce FinCDM, the first cognitive
diagnosis evaluation framework tailored for financial LLMs, enabling the
evaluation of LLMs at the knowledge-skill level, identifying what financial
skills and knowledge they have or lack based on their response patterns across
skill-tagged tasks, rather than a single aggregated number. We construct
CPA-QKA, the first cognitively informed financial evaluation dataset derived
from the Certified Public Accountant (CPA) examination, with comprehensive
coverage of real-world accounting and financial skills. It is rigorously
annotated by domain experts, who author, validate, and annotate questions with
high inter-annotator agreement and fine-grained knowledge labels. Our extensive
experiments on 30 proprietary, open-source, and domain-specific LLMs show that
FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax
and regulatory reasoning overlooked by traditional benchmarks, and uncovers
behavioral clusters among models. FinCDM introduces a new paradigm for
financial LLM evaluation by enabling interpretable, skill-aware diagnosis that
supports more trustworthy and targeted model development, and all datasets and
evaluation scripts will be publicly released to support further research.
68a685519e4b49496aac6938
Ziyan Kuang
68a685519e4b49496aac6939
Feiyu Zhu
68a685519e4b49496aac693a
Maowei Jiang
68a685519e4b49496aac693b
Yanzhao Lai
admin_assigned
2025-08-21T09:58:05.932Z
6880eca712035d59e50ba716
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-6sPgohq0n_-oxFmyAvmr.png
Yanzhao Lai
user
2083L
68a685519e4b49496aac693c
Zelin Wang
68a685519e4b49496aac693d
Zhitong Wang
68a685519e4b49496aac693e
Meikang Qiu
68a685519e4b49496aac693f
Jiajia Huang
admin_assigned
2025-08-21T09:57:35.753Z
66f3c0f97a2d332b0d81d18f
/avatars/a8283846b8b104af0c2d3566455b2004.svg
Jiajia Huang
user
hugai101
68a685519e4b49496aac6940
Min Peng
68a685519e4b49496aac6941
Qianqian Xie
admin_assigned
2025-08-21T09:57:19.006Z
6479f4317c18dca75e9a9324
/avatars/9aa709230b057f57ee4415c04a622c63.svg
Xie
user
QianqianXie1994
68a685519e4b49496aac6942
Sophia Ananiadou
admin_assigned
2025-08-21T09:57:04.485Z
66f6cb352c5d4ef3578a9c3f
/avatars/0a70c94072bc5e1d018cf12da0904ff0.svg
Sophia Ananiadou
user
Effoula
From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating
  Financial Large Language Models
Large Language Models (LLMs) have shown promise for financial applications,
yet their suitability for this high-stakes domain remains largely unproven due
to inadequacies in existing benchmarks. Existing benchmarks solely rely on
score-level evaluation, summarizing performance with a single score that
obscures the nuanced understanding of what models truly know and their precise
limitations. They also rely on datasets that cover only a narrow subset of
financial concepts, while overlooking other essentials for real-world
applications. To address these gaps, we introduce FinCDM, the first cognitive
diagnosis evaluation framework tailored for financial LLMs, enabling the
evaluation of LLMs at the knowledge-skill level, identifying what financial
skills and knowledge they have or lack based on their response patterns across
skill-tagged tasks, rather than a single aggregated number. We construct
CPA-QKA, the first cognitively informed financial evaluation dataset derived
from the Certified Public Accountant (CPA) examination, with comprehensive
coverage of real-world accounting and financial skills. It is rigorously
annotated by domain experts, who author, validate, and annotate questions with
high inter-annotator agreement and fine-grained knowledge labels. Our extensive
experiments on 30 proprietary, open-source, and domain-specific LLMs show that
FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax
and regulatory reasoning overlooked by traditional benchmarks, and uncovers
behavioral clusters among models. FinCDM introduces a new paradigm for
financial LLM evaluation by enabling interpretable, skill-aware diagnosis that
supports more trustworthy and targeted model development, and all datasets and
evaluation scripts will be publicly released to support further research.
FinCDM, a cognitive diagnosis framework, evaluates financial LLMs at the knowledge-skill level using a comprehensive dataset, revealing hidden knowledge gaps and supporting more trustworthy model development.
Large Language Models FinCDM cognitive diagnosis evaluation framework CPA-QKA Certified Public Accountant accounting and financial skills knowledge-skill level knowledge gaps tax and regulatory reasoning behavioral clusters skill-aware diagnosis