Inside-Out: Hidden Factual Knowledge in LLMs
This work presents a framework for assessing whether large language models
(LLMs) encode more factual knowledge in their parameters than what they express
in their outputs. While a few studies hint at this possibility, none has
clearly defined or demonstrated this phenomenon. We first propose a formal
definition of knowledge, quantifying it for a given question as the fraction of
correct-incorrect answer pairs where the correct one is ranked higher. This
gives rise to external and internal knowledge, depending on the information
used to score individual answer candidates: either the model's observable
token-level probabilities or its intermediate computations. Hidden knowledge
arises when internal knowledge exceeds external knowledge. We then present a
case study, applying this framework to three popular open-weights LLMs in a
closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
more factual knowledge internally than what they express externally, with an
average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
model can internally know an answer perfectly, yet fail to generate it even
once, despite large-scale repeated sampling of 1,000 answers. This reveals
fundamental limitations in the generation capabilities of LLMs, which (3) puts
a practical constraint on scaling test-time compute via repeated answer
sampling in closed-book QA: significant performance improvements remain
inaccessible because some answers are practically never sampled, yet if they
were, we would be guaranteed to rank them first.
67dd5e658dc9e947cee90f4a
Zorik Gekhman
extracted_confirmed
2025-03-21T13:57:32.354Z
61499a8f8517f77f09a3a990
/avatars/fa15d117e8ce8c50669455d3e483408b.svg
Zorik
user
zorik
67dd5e658dc9e947cee90f4b
Eyal Ben David
67dd5e658dc9e947cee90f4c
Hadas Orgad
67dd5e658dc9e947cee90f4d
Eran Ofek
67dd5e658dc9e947cee90f4e
Yonatan Belinkov
67dd5e658dc9e947cee90f4f
Idan Szpector
67dd5e658dc9e947cee90f50
Jonathan Herzig
67dd5e658dc9e947cee90f51
Roi Reichart
Inside-Out: Hidden Factual Knowledge in LLMs
This work presents a framework for assessing whether large language models
(LLMs) encode more factual knowledge in their parameters than what they express
in their outputs. While a few studies hint at this possibility, none has
clearly defined or demonstrated this phenomenon. We first propose a formal
definition of knowledge, quantifying it for a given question as the fraction of
correct-incorrect answer pairs where the correct one is ranked higher. This
gives rise to external and internal knowledge, depending on the information
used to score individual answer candidates: either the model's observable
token-level probabilities or its intermediate computations. Hidden knowledge
arises when internal knowledge exceeds external knowledge. We then present a
case study, applying this framework to three popular open-weights LLMs in a
closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
more factual knowledge internally than what they express externally, with an
average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
model can internally know an answer perfectly, yet fail to generate it even
once, despite large-scale repeated sampling of 1,000 answers. This reveals
fundamental limitations in the generation capabilities of LLMs, which (3) puts
a practical constraint on scaling test-time compute via repeated answer
sampling in closed-book QA: significant performance improvements remain
inaccessible because some answers are practically never sampled, yet if they
were, we would be guaranteed to rank them first.
large language models (LLMs) knowledge correct-incorrect answer pairs external knowledge internal knowledge token-level probabilities intermediate computations hidden knowledge closed-book QA repeated sampling generation capabilities