Discovering Knowledge Deficiencies of Language Models on Massive
  Knowledge Base
Large language models (LLMs) possess impressive linguistic capabilities but
often fail to faithfully retain factual knowledge, leading to hallucinations
and unreliable outputs. Understanding LLMs' knowledge deficiencies by
exhaustively evaluating against full-scale knowledge bases is computationally
prohibitive, especially for closed-weight models. We propose stochastic error
ascent (SEA), a scalable and efficient framework for discovering knowledge
deficiencies (errors) in closed-weight LLMs under a strict query budget. Rather
than naively probing all knowledge candidates, SEA formulates error discovery
as a stochastic optimization process: it iteratively retrieves new high-error
candidates by leveraging the semantic similarity to previously observed
failures. To further enhance search efficiency and coverage, SEA employs
hierarchical retrieval across document and paragraph levels, and constructs a
relation directed acyclic graph to model error propagation and identify
systematic failure modes. Empirically, SEA uncovers 40.7x more knowledge errors
than Automated Capability Discovery and 26.7% more than AutoBencher, while
reducing the cost-per-error by 599x and 9x, respectively. Human evaluation
confirms the high quality of generated questions, while ablation and
convergence analyses validate the contribution of each component in SEA.
Further analysis on the discovered errors reveals correlated failure patterns
across LLM families and recurring deficits, highlighting the need for better
data coverage and targeted fine-tuning in future LLM development.
67eb57a56522661171fb4725
Linxin Song
claimed_verified
2025-04-02T08:23:45.180Z
64d660308ebc40443813f014
/avatars/516bb2d2383be99794e366dfb41636b6.svg
Linxin Song
user
linxinso
67eb57a56522661171fb4726
Xuwei Ding
admin_assigned
2025-04-02T12:26:16.488Z
676f6c0ccb094bb8d8a0948f
/avatars/78d26c39e1108ee4029639e4202d0ec9.svg
Xuwei Ding
user
Xuwei04
67eb57a56522661171fb4727
Jieyu Zhang
admin_assigned
2025-04-02T12:26:35.397Z
640131b08ba76abe4b71b5d0
/avatars/2288b96a9a0ae8f584768f54e098def1.svg
Jieyu Zhang
user
jieyuz2
67eb57a56522661171fb4728
Taiwei Shi
claimed_verified
2025-04-01T07:46:48.349Z
62e1b3cb3eb0730f621a83f6
https://cdn-avatars.huggingface.co/v1/production/uploads/1658958764563-noauth.jpeg
Taiwei Shi
user
MaksimSTW
67eb57a56522661171fb4729
Ryotaro Shimizu
admin_assigned
2025-04-02T12:26:41.560Z
65641c8ec683336b76fecb7f
/avatars/3dcf51fc55f7e5ce2c9e84eaee270277.svg
Ryotaro Shimizu
user
latataro
67eb57a56522661171fb472a
Rahul Gupta
67eb57a56522661171fb472b
Yang Liu
67eb57a56522661171fb472c
Jian Kang
67eb57a56522661171fb472d
Jieyu Zhao
admin_assigned
2025-04-02T12:27:18.161Z
64ba0230c9a9894feb0cd32b
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba0230c9a9894feb0cd32b/V1XWySRXAhLN4wqXZJ3Ai.jpeg
Jieyu Zhao
user
jieyuz
Discovering Knowledge Deficiencies of Language Models on Massive
  Knowledge Base
Large language models (LLMs) possess impressive linguistic capabilities but
often fail to faithfully retain factual knowledge, leading to hallucinations
and unreliable outputs. Understanding LLMs' knowledge deficiencies by
exhaustively evaluating against full-scale knowledge bases is computationally
prohibitive, especially for closed-weight models. We propose stochastic error
ascent (SEA), a scalable and efficient framework for discovering knowledge
deficiencies (errors) in closed-weight LLMs under a strict query budget. Rather
than naively probing all knowledge candidates, SEA formulates error discovery
as a stochastic optimization process: it iteratively retrieves new high-error
candidates by leveraging the semantic similarity to previously observed
failures. To further enhance search efficiency and coverage, SEA employs
hierarchical retrieval across document and paragraph levels, and constructs a
relation directed acyclic graph to model error propagation and identify
systematic failure modes. Empirically, SEA uncovers 40.7x more knowledge errors
than Automated Capability Discovery and 26.7% more than AutoBencher, while
reducing the cost-per-error by 599x and 9x, respectively. Human evaluation
confirms the high quality of generated questions, while ablation and
convergence analyses validate the contribution of each component in SEA.
Further analysis on the discovered errors reveals correlated failure patterns
across LLM families and recurring deficits, highlighting the need for better
data coverage and targeted fine-tuning in future LLM development.
Stochastic Error Ascent (SEA) is a scalable framework that identifies knowledge errors in large language models using semantic similarity and hierarchical retrieval, uncovering more errors with lower costs compared to existing methods.
stochastic error ascent SEA knowledge deficiencies closed-weight LLMs semantic similarity hierarchical retrieval document level paragraph level relation directed acyclic graph error propagation systematic failure modes