Inside-Out: Hidden Factual Knowledge in LLMs
This work presents a framework for assessing whether large language models
(LLMs) encode more factual knowledge in their parameters than what they express
in their outputs. While a few studies hint at this possibility, none has
clearly defined or demonstrated this phenomenon. We first propose a formal
definition of knowledge, quantifying it for a given question as the fraction of
correct-incorrect answer pairs where the correct one is ranked higher. This
gives rise to external and internal knowledge, depending on the information
used to score individual answer candidates: either the model's observable
token-level probabilities or its intermediate computations. Hidden knowledge
arises when internal knowledge exceeds external knowledge. We then present a
case study, applying this framework to three popular open-weights LLMs in a
closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
more factual knowledge internally than what they express externally, with an
average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
model can internally know an answer perfectly, yet fail to generate it even
once, despite large-scale repeated sampling of 1,000 answers. This reveals
fundamental limitations in the generation capabilities of LLMs, which (3) puts
a practical constraint on scaling test-time compute via repeated answer
sampling in closed-book QA: significant performance improvements remain
inaccessible because some answers are practically never sampled, yet if they
were, we would be guaranteed to rank them first.
67dd5e658dc9e947cee90f4a
Zorik Gekhman
extracted_confirmed
2025-03-21T13:57:32.354Z
61499a8f8517f77f09a3a990
/avatars/fa15d117e8ce8c50669455d3e483408b.svg
Zorik
user
zorik
67dd5e658dc9e947cee90f4b
Eyal Ben David
67dd5e658dc9e947cee90f4c
Hadas Orgad
67dd5e658dc9e947cee90f4d
Eran Ofek
67dd5e658dc9e947cee90f4e
Yonatan Belinkov
67dd5e658dc9e947cee90f4f
Idan Szpector
67dd5e658dc9e947cee90f50
Jonathan Herzig
67dd5e658dc9e947cee90f51
Roi Reichart
Inside-Out: Hidden Factual Knowledge in LLMs
This work presents a framework for assessing whether large language models
(LLMs) encode more factual knowledge in their parameters than what they express
in their outputs. While a few studies hint at this possibility, none has
clearly defined or demonstrated this phenomenon. We first propose a formal
definition of knowledge, quantifying it for a given question as the fraction of
correct-incorrect answer pairs where the correct one is ranked higher. This
gives rise to external and internal knowledge, depending on the information
used to score individual answer candidates: either the model's observable
token-level probabilities or its intermediate computations. Hidden knowledge
arises when internal knowledge exceeds external knowledge. We then present a
case study, applying this framework to three popular open-weights LLMs in a
closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
more factual knowledge internally than what they express externally, with an
average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
model can internally know an answer perfectly, yet fail to generate it even
once, despite large-scale repeated sampling of 1,000 answers. This reveals
fundamental limitations in the generation capabilities of LLMs, which (3) puts
a practical constraint on scaling test-time compute via repeated answer
sampling in closed-book QA: significant performance improvements remain
inaccessible because some answers are practically never sampled, yet if they
were, we would be guaranteed to rank them first.
LLMs encode more internal factual knowledge than they express externally, with some knowledge so deeply hidden that it is never generated, despite repeated sampling.
large language models LLMs knowledge token-level probabilities intermediate computations external knowledge internal knowledge hidden knowledge closed-book QA generation capabilities