LMEnt: A Suite for Analyzing Knowledge in Language Models from
  Pretraining Data to Representations
Language models (LMs) increasingly drive real-world applications that require
world knowledge. However, the internal processes through which models turn data
into representations of knowledge and beliefs about the world, are poorly
understood. Insights into these processes could pave the way for developing LMs
with knowledge representations that are more consistent, robust, and complete.
To facilitate studying these questions, we present LMEnt, a suite for analyzing
knowledge acquisition in LMs during pretraining. LMEnt introduces: (1) a
knowledge-rich pretraining corpus, fully annotated with entity mentions, based
on Wikipedia, (2) an entity-based retrieval method over pretraining data that
outperforms previous approaches by as much as 80.4%, and (3) 12 pretrained
models with up to 1B parameters and 4K intermediate checkpoints, with
comparable performance to popular open-sourced models on knowledge benchmarks.
Together, these resources provide a controlled environment for analyzing
connections between entity mentions in pretraining and downstream performance,
and the effects of causal interventions in pretraining data. We show the
utility of LMEnt by studying knowledge acquisition across checkpoints, finding
that fact frequency is key, but does not fully explain learning trends. We
release LMEnt to support studies of knowledge in LMs, including knowledge
representations, plasticity, editing, attribution, and learning dynamics.
68b94c5ed43cadaf7a688c0a
Daniela Gottesman
68b94c5ed43cadaf7a688c0b
Alon Gilae-Dotan
68b94c5ed43cadaf7a688c0c
Ido Cohen
68b94c5ed43cadaf7a688c0d
Yoav Gur-Arieh
68b94c5ed43cadaf7a688c0e
Marius Mosbach
68b94c5ed43cadaf7a688c0f
Ori Yoran
68b94c5ed43cadaf7a688c10
Mor Geva
claimed_verified
2025-09-04T08:42:49.827Z
610b729f9da682cd54ad9adf
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
Mor Geva
user
mega
LMEnt: A Suite for Analyzing Knowledge in Language Models from
  Pretraining Data to Representations
Language models (LMs) increasingly drive real-world applications that require
world knowledge. However, the internal processes through which models turn data
into representations of knowledge and beliefs about the world, are poorly
understood. Insights into these processes could pave the way for developing LMs
with knowledge representations that are more consistent, robust, and complete.
To facilitate studying these questions, we present LMEnt, a suite for analyzing
knowledge acquisition in LMs during pretraining. LMEnt introduces: (1) a
knowledge-rich pretraining corpus, fully annotated with entity mentions, based
on Wikipedia, (2) an entity-based retrieval method over pretraining data that
outperforms previous approaches by as much as 80.4%, and (3) 12 pretrained
models with up to 1B parameters and 4K intermediate checkpoints, with
comparable performance to popular open-sourced models on knowledge benchmarks.
Together, these resources provide a controlled environment for analyzing
connections between entity mentions in pretraining and downstream performance,
and the effects of causal interventions in pretraining data. We show the
utility of LMEnt by studying knowledge acquisition across checkpoints, finding
that fact frequency is key, but does not fully explain learning trends. We
release LMEnt to support studies of knowledge in LMs, including knowledge
representations, plasticity, editing, attribution, and learning dynamics.
LMEnt is a suite for analyzing knowledge acquisition in language models during pretraining, providing annotated corpora, retrieval methods, and pretrained models to study knowledge representations and learning dynamics.
language models knowledge acquisition pretraining knowledge-rich pretraining corpus entity mentions entity-based retrieval pretrained models knowledge benchmarks knowledge representations knowledge plasticity knowledge editing knowledge attribution learning dynamics