Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for
  Real-world Knowledge Injection
Despite significant advances in large language models (LLMs), their knowledge
memorization capabilities remain underexplored, due to the lack of standardized
and high-quality test ground. In this paper, we introduce a novel, real-world
and large-scale knowledge injection benchmark that evolves continuously over
time without requiring human intervention. Specifically, we propose WikiDYK,
which leverages recently-added and human-written facts from Wikipedia's "Did
You Know..." entries. These entries are carefully selected by expert Wikipedia
editors based on criteria such as verifiability and clarity. Each entry is
converted into multiple question-answer pairs spanning diverse task formats
from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290
facts and 77,180 questions, which is also seamlessly extensible with future
updates from Wikipedia editors. Extensive experiments using continued
pre-training reveal a surprising insight: despite their prevalence in modern
LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge
memorization capabilities compared to Bidirectional Language Models (BiLMs),
exhibiting a 23% lower accuracy in terms of reliability. To compensate for the
smaller scales of current BiLMs, we introduce a modular collaborative framework
utilizing ensembles of BiLMs as external knowledge repositories to integrate
with LLMs. Experiment shows that our framework further improves the reliability
accuracy by up to 29.1%.
682d2f9f3b5f51f42185dd7d
Yuwei Zhang
682d2f9f3b5f51f42185dd7e
Wenhao Yu
682d2f9f3b5f51f42185dd7f
Shangbin Feng
682d2f9f3b5f51f42185dd80
Yifan Zhu
682d2f9f3b5f51f42185dd81
Letian Peng
admin_assigned
2025-05-21T10:21:33.327Z
64323dd503d81fa4d26deaf9
https://cdn-avatars.huggingface.co/v1/production/uploads/64323dd503d81fa4d26deaf9/x3ES8VXEZJljxDWvFWaAf.png
Letian Peng
user
KomeijiForce
682d2f9f3b5f51f42185dd82
Jayanth Srinivasa
682d2f9f3b5f51f42185dd83
Gaowen Liu
682d2f9f3b5f51f42185dd84
Jingbo Shang
admin_assigned
2025-05-21T10:21:40.883Z
660655119e3555d648f6c6b5
/avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
Jingbo Shang
user
shangjingbo
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for
  Real-world Knowledge Injection
Despite significant advances in large language models (LLMs), their knowledge
memorization capabilities remain underexplored, due to the lack of standardized
and high-quality test ground. In this paper, we introduce a novel, real-world
and large-scale knowledge injection benchmark that evolves continuously over
time without requiring human intervention. Specifically, we propose WikiDYK,
which leverages recently-added and human-written facts from Wikipedia's "Did
You Know..." entries. These entries are carefully selected by expert Wikipedia
editors based on criteria such as verifiability and clarity. Each entry is
converted into multiple question-answer pairs spanning diverse task formats
from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290
facts and 77,180 questions, which is also seamlessly extensible with future
updates from Wikipedia editors. Extensive experiments using continued
pre-training reveal a surprising insight: despite their prevalence in modern
LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge
memorization capabilities compared to Bidirectional Language Models (BiLMs),
exhibiting a 23% lower accuracy in terms of reliability. To compensate for the
smaller scales of current BiLMs, we introduce a modular collaborative framework
utilizing ensembles of BiLMs as external knowledge repositories to integrate
with LLMs. Experiment shows that our framework further improves the reliability
accuracy by up to 29.1%.
A new knowledge injection benchmark, WikiDYK, highlights differences in knowledge memorization between CLMs and BiLMs, and a modular collaborative framework enhances LLM reliability.
large language models knowledge injection benchmark Causal Language Models Bidirectional Language Models knowledge memorization WikiDYK modular collaborative framework