Does Circuit Analysis Interpretability Scale? Evidence from Multiple
  Choice Capabilities in Chinchilla
Circuit analysis is a promising technique for understanding the
internal mechanisms of language models. However, existing analyses are done in
small models far from the state of the art. To address this, we present a case
study of circuit analysis in the 70B Chinchilla model, aiming to test the
scalability of circuit analysis. In particular, we study multiple-choice
question answering, and investigate Chinchilla's capability to identify the
correct answer label given knowledge of the correct answer text.
We find that the existing techniques of logit attribution, attention pattern
visualization, and activation patching naturally scale to Chinchilla, allowing
us to identify and categorize a small set of `output nodes' (attention heads
and MLPs).
  We further study the `correct letter' category of attention heads aiming to
understand the semantics of their features, with mixed results. For normal
multiple-choice question answers, we significantly compress the query, key and
value subspaces of the head without loss of performance when operating on the
answer labels for multiple-choice questions, and we show that the query and key
subspaces represent an `Nth item in an enumeration' feature to at least some
extent. However, when we attempt to use this explanation to understand the
heads' behaviour on a more general distribution including randomized answer
labels, we find that it is only a partial explanation, suggesting there is more
to learn about the operation of `correct letter' heads on multiple choice
question answering.
64b7476c9f5987572c9b2cfa
Tom Lieberum
claimed_verified
2023-07-20T07:17:40.948Z
632cb97554e2c512c8f7bb31
/avatars/634be7f8a59b6cf639e9e6c5e7ca1fbd.svg
TomL
user
Aric
64b7476c9f5987572c9b2cfb
Matthew Rahtz
64b7476c9f5987572c9b2cfc
János Kramár
64b7476c9f5987572c9b2cfd
Geoffrey Irving
64b7476c9f5987572c9b2cfe
Rohin Shah
64b7476c9f5987572c9b2cff
Vladimir Mikulik
Does Circuit Analysis Interpretability Scale? Evidence from Multiple
  Choice Capabilities in Chinchilla
Circuit analysis is a promising technique for understanding the
internal mechanisms of language models. However, existing analyses are done in
small models far from the state of the art. To address this, we present a case
study of circuit analysis in the 70B Chinchilla model, aiming to test the
scalability of circuit analysis. In particular, we study multiple-choice
question answering, and investigate Chinchilla's capability to identify the
correct answer label given knowledge of the correct answer text.
We find that the existing techniques of logit attribution, attention pattern
visualization, and activation patching naturally scale to Chinchilla, allowing
us to identify and categorize a small set of `output nodes' (attention heads
and MLPs).
  We further study the `correct letter' category of attention heads aiming to
understand the semantics of their features, with mixed results. For normal
multiple-choice question answers, we significantly compress the query, key and
value subspaces of the head without loss of performance when operating on the
answer labels for multiple-choice questions, and we show that the query and key
subspaces represent an `Nth item in an enumeration' feature to at least some
extent. However, when we attempt to use this explanation to understand the
heads' behaviour on a more general distribution including randomized answer
labels, we find that it is only a partial explanation, suggesting there is more
to learn about the operation of `correct letter' heads on multiple choice
question answering.
Circuit analysis applied to a 70B Chinchilla model provides insights into the mechanisms of multiple-choice question answering, demonstrating that logit attribution, attention pattern visualization, and activation patching scale and allow identification of output nodes like attention heads and MLPs.
logit attribution attention pattern visualization activation patching attention heads MLPs query subspace key subspace value subspace enumeration feature