FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark
  for Evaluating LLMs
The complexity of the Generally Accepted Accounting Principles (GAAP) and the
hierarchical structure of eXtensible Business Reporting Language (XBRL) filings
make financial auditing increasingly difficult to automate and verify. While
large language models (LLMs) have demonstrated strong capabilities in
unstructured text understanding, their ability to reason over structured,
interdependent, and taxonomy-driven financial documents remains largely
unexplored. To fill this gap, we introduce FinAuditing, the first
taxonomy-aligned, structure-aware, multi-document benchmark for evaluating LLMs
on financial auditing tasks. Built from real US-GAAP-compliant XBRL filings,
FinAuditing defines three complementary subtasks, FinSM for semantic
consistency, FinRE for relational consistency, and FinMR for numerical
consistency, each targeting a distinct aspect of structured auditing reasoning.
We further propose a unified evaluation framework integrating retrieval,
classification, and reasoning metrics across these subtasks. Extensive
zero-shot experiments on 13 state-of-the-art LLMs reveal that current models
perform inconsistently across semantic, relational, and mathematical
dimensions, with accuracy drops of up to 60-90% when reasoning over
hierarchical multi-document structures. Our findings expose the systematic
limitations of modern LLMs in taxonomy-grounded financial reasoning and
establish FinAuditing as a foundation for developing trustworthy,
structure-aware, and regulation-aligned financial intelligence systems. The
benchmark dataset is available at Hugging Face.
68edac0dde1fee572713a705
Yan Wang
claimed_verified
2025-10-14T07:30:56.790Z
65d76cc5b9b7b8bf88faa916
/avatars/d95232cd0c307efab6197ade1a66190b.svg
Yan Wang
user
YanAdjeNole
68edac0dde1fee572713a706
Keyi Wang
68edac0dde1fee572713a707
Shanshan Yang
68edac0dde1fee572713a708
Jaisal Patel
68edac0dde1fee572713a709
Jeff Zhao
68edac0dde1fee572713a70a
Fengran Mo
68edac0dde1fee572713a70b
Xueqing Peng
68edac0dde1fee572713a70c
Lingfei Qian
68edac0dde1fee572713a70d
Jimin Huang
claimed_verified
2025-10-14T07:30:58.956Z
63b58ed5889aa6707f0bb0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
Jimin Huang
user
jiminHuang
68edac0dde1fee572713a70e
Guojun Xiong
68edac0dde1fee572713a70f
Xiao-Yang Liu
68edac0dde1fee572713a710
Jian-Yun Nie
FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark
  for Evaluating LLMs
The complexity of the Generally Accepted Accounting Principles (GAAP) and the
hierarchical structure of eXtensible Business Reporting Language (XBRL) filings
make financial auditing increasingly difficult to automate and verify. While
large language models (LLMs) have demonstrated strong capabilities in
unstructured text understanding, their ability to reason over structured,
interdependent, and taxonomy-driven financial documents remains largely
unexplored. To fill this gap, we introduce FinAuditing, the first
taxonomy-aligned, structure-aware, multi-document benchmark for evaluating LLMs
on financial auditing tasks. Built from real US-GAAP-compliant XBRL filings,
FinAuditing defines three complementary subtasks, FinSM for semantic
consistency, FinRE for relational consistency, and FinMR for numerical
consistency, each targeting a distinct aspect of structured auditing reasoning.
We further propose a unified evaluation framework integrating retrieval,
classification, and reasoning metrics across these subtasks. Extensive
zero-shot experiments on 13 state-of-the-art LLMs reveal that current models
perform inconsistently across semantic, relational, and mathematical
dimensions, with accuracy drops of up to 60-90% when reasoning over
hierarchical multi-document structures. Our findings expose the systematic
limitations of modern LLMs in taxonomy-grounded financial reasoning and
establish FinAuditing as a foundation for developing trustworthy,
structure-aware, and regulation-aligned financial intelligence systems. The
benchmark dataset is available at Hugging Face.
FinAuditing is a benchmark for evaluating LLMs on structured financial auditing tasks, revealing their limitations in handling taxonomy-driven, hierarchical financial documents.
LLMs FinAuditing taxonomy-aligned structure-aware multi-document benchmark FinSM FinRE FinMR semantic consistency relational consistency numerical consistency retrieval classification reasoning metrics zero-shot experiments US-GAAP XBRL filings