Challenges with unsupervised LLM knowledge discovery
We show that existing unsupervised methods on large language model (LLM)
activations do not discover knowledge -- instead they seem to discover whatever
feature of the activations is most prominent. The idea behind unsupervised
knowledge elicitation is that knowledge satisfies a consistency structure,
which can be used to discover knowledge. We first prove theoretically that
arbitrary features (not just knowledge) satisfy the consistency structure of a
particular leading unsupervised knowledge-elicitation method,
contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a
series of experiments showing settings in which unsupervised methods result in
classifiers that do not predict knowledge, but instead predict a different
prominent feature. We conclude that existing unsupervised methods for
discovering latent knowledge are insufficient, and we contribute sanity checks
to apply to evaluating future knowledge elicitation methods. Conceptually, we
hypothesise that the identification issues explored here, e.g. distinguishing a
model's knowledge from that of a simulated character's, will persist for future
unsupervised methods.
657fb5e79a9e347d4bf6378d
Sebastian Farquhar
657fb5e79a9e347d4bf6378e
Vikrant Varma
657fb5e79a9e347d4bf6378f
Zachary Kenton
657fb5e79a9e347d4bf63790
Johannes Gasteiger
657fb5e79a9e347d4bf63791
Vladimir Mikulik
657fb5e79a9e347d4bf63792
Rohin Shah
Challenges with unsupervised LLM knowledge discovery
We show that existing unsupervised methods on large language model (LLM)
activations do not discover knowledge -- instead they seem to discover whatever
feature of the activations is most prominent. The idea behind unsupervised
knowledge elicitation is that knowledge satisfies a consistency structure,
which can be used to discover knowledge. We first prove theoretically that
arbitrary features (not just knowledge) satisfy the consistency structure of a
particular leading unsupervised knowledge-elicitation method,
contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a
series of experiments showing settings in which unsupervised methods result in
classifiers that do not predict knowledge, but instead predict a different
prominent feature. We conclude that existing unsupervised methods for
discovering latent knowledge are insufficient, and we contribute sanity checks
to apply to evaluating future knowledge elicitation methods. Conceptually, we
hypothesise that the identification issues explored here, e.g. distinguishing a
model's knowledge from that of a simulated character's, will persist for future
unsupervised methods.
Unsupervised methods on large language model activations do not consistently discover knowledge and instead predict prominent features, indicating a need for improved evaluation methodologies.
large language model LLM activations unsupervised knowledge elicitation consistency structure contrast-consistent search classifiers