Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
Obtaining human-interpretable explanations of large, general-purpose language
models is an urgent goal for AI safety. However, it is just as important that
our interpretability methods are faithful to the causal dynamics underlying
model behavior and able to robustly generalize to unseen inputs. Distributed
Alignment Search (DAS) is a powerful gradient descent method grounded in a
theory of causal abstraction that uncovered perfect alignments between
interpretable symbolic algorithms and small deep learning models fine-tuned for
specific tasks. In the present paper, we scale DAS significantly by replacing
the remaining brute-force search steps with learned parameters -- an approach
we call DAS. This enables us to efficiently search for interpretable causal
structure in large language models while they follow instructions. We apply DAS
to the Alpaca model (7B parameters), which, off the shelf, solves a simple
numerical reasoning problem. With DAS, we discover that Alpaca does this by
implementing a causal model with two interpretable boolean variables.
Furthermore, we find that the alignment of neural representations with these
variables is robust to changes in inputs and instructions. These findings mark
a first step toward deeply understanding the inner-workings of our largest and
most widely deployed language models.
6462f224d1ccd517f46a807f
Zhengxuan Wu
claimed_verified
2023-05-17T08:04:16.333Z
604fa06d4f9b833fc41c75cf
https://cdn-avatars.huggingface.co/v1/production/uploads/1650432433568-604fa06d4f9b833fc41c75cf.png
Zhengxuan Wu
user
zhengxuanzenwu
6462f224d1ccd517f46a8080
Atticus Geiger
claimed_verified
2023-05-16T15:31:48.092Z
627b2d0527dc4650b62eef42
/avatars/e70381850f5657b54e90f5539f3d74eb.svg
Atticus Geiger
user
atticusg
6462f224d1ccd517f46a8081
Christopher Potts
claimed_verified
2023-05-16T16:22:39.048Z
62606790e3dd8b49f360f050
https://cdn-avatars.huggingface.co/v1/production/uploads/1650485126923-noauth.png
Chris Potts
user
cgpotts
6462f224d1ccd517f46a8082
Noah D. Goodman
claimed_verified
2024-11-15T09:57:48.804Z
67321274c1f20c742bcf7a8d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
Noah D. Goodman
user
ngoodman
Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
Obtaining human-interpretable explanations of large, general-purpose language
models is an urgent goal for AI safety. However, it is just as important that
our interpretability methods are faithful to the causal dynamics underlying
model behavior and able to robustly generalize to unseen inputs. Distributed
Alignment Search (DAS) is a powerful gradient descent method grounded in a
theory of causal abstraction that uncovered perfect alignments between
interpretable symbolic algorithms and small deep learning models fine-tuned for
specific tasks. In the present paper, we scale DAS significantly by replacing
the remaining brute-force search steps with learned parameters -- an approach
we call DAS. This enables us to efficiently search for interpretable causal
structure in large language models while they follow instructions. We apply DAS
to the Alpaca model (7B parameters), which, off the shelf, solves a simple
numerical reasoning problem. With DAS, we discover that Alpaca does this by
implementing a causal model with two interpretable boolean variables.
Furthermore, we find that the alignment of neural representations with these
variables is robust to changes in inputs and instructions. These findings mark
a first step toward deeply understanding the inner-workings of our largest and
most widely deployed language models.
DAS, an enhanced gradient descent method, identifies interpretable causal structures in large language models like Alpaca by aligning neural representations with symbolic boolean variables.
Distributed Alignment Search (DAS) gradient descent causal abstraction interpretable symbolic algorithms fine-tuned models causal structure large language models Alpaca model numerical reasoning problem boolean variables neural representations