Explaining black box text modules in natural language with language
  models
Large language models (LLMs) have demonstrated remarkable prediction
performance for a growing array of tasks. However, their rapid proliferation
and increasing opaqueness have created a growing need for interpretability.
Here, we ask whether we can automatically obtain natural language explanations
for black box text modules. A "text module" is any function that maps text to a
scalar continuous value, such as a submodule within an LLM or a fitted model of
a brain region. "Black box" indicates that we only have access to the module's
inputs/outputs.
  We introduce Summarize and Score (SASC), a method that takes in a text module
and returns a natural language explanation of the module's selectivity along
with a score for how reliable the explanation is. We study SASC in 3 contexts.
First, we evaluate SASC on synthetic modules and find that it often recovers
ground truth explanations. Second, we use SASC to explain modules found within
a pre-trained BERT model, enabling inspection of the model's internals.
Finally, we show that SASC can generate explanations for the response of
individual fMRI voxels to language stimuli, with potential applications to
fine-grained brain mapping. All code for using SASC and reproducing results is
made available on Github.
646585af5cdb9ab50f874bbe
Chandan Singh
claimed_verified
2023-07-13T08:33:11.745Z
62b0bee117fa24819dd0999c
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0bee117fa24819dd0999c/eJ7ryFQ3_ZbqV29qf4KyM.jpeg
Chandan Singh
user
csinva
646585af5cdb9ab50f874bbf
Aliyah R. Hsu
646585af5cdb9ab50f874bc0
Richard Antonello
claimed_verified
2023-05-22T09:36:57.674Z
646b354e36505117e229b773
/avatars/3a8c51bcb969585502667faf7a7b6496.svg
Richard Antonello
user
RAntonello1
646585af5cdb9ab50f874bc1
Shailee Jain
claimed_verified
2023-05-19T19:04:16.631Z
62b2a167e567c41e4a1433cb
/avatars/584f58bad340ab1418f4923fd18bc0d0.svg
Shailee Jain
user
shailee
646585af5cdb9ab50f874bc2
Alexander G. Huth
claimed_verified
2023-05-22T18:01:35.895Z
6303d6633926de1f7ec2fc42
/avatars/4ce51df91ed381fd0b1f4d8f46b1e65a.svg
Alex Huth
user
alexhuth
646585af5cdb9ab50f874bc3
Bin Yu
claimed_verified
2023-05-19T19:04:19.932Z
64679498374fe5728d4044d5
/avatars/78516e1c9532900cce19f81483855db8.svg
Bin Yu
user
binyu000
646585af5cdb9ab50f874bc4
Jianfeng Gao
Explaining black box text modules in natural language with language
  models
Large language models (LLMs) have demonstrated remarkable prediction
performance for a growing array of tasks. However, their rapid proliferation
and increasing opaqueness have created a growing need for interpretability.
Here, we ask whether we can automatically obtain natural language explanations
for black box text modules. A "text module" is any function that maps text to a
scalar continuous value, such as a submodule within an LLM or a fitted model of
a brain region. "Black box" indicates that we only have access to the module's
inputs/outputs.
  We introduce Summarize and Score (SASC), a method that takes in a text module
and returns a natural language explanation of the module's selectivity along
with a score for how reliable the explanation is. We study SASC in 3 contexts.
First, we evaluate SASC on synthetic modules and find that it often recovers
ground truth explanations. Second, we use SASC to explain modules found within
a pre-trained BERT model, enabling inspection of the model's internals.
Finally, we show that SASC can generate explanations for the response of
individual fMRI voxels to language stimuli, with potential applications to
fine-grained brain mapping. All code for using SASC and reproducing results is
made available on Github.
SASC, a method for generating natural language explanations for black-box text modules, demonstrates reliability in synthetic, BERT model, and fMRI voxel contexts.
large language models synthetic modules Summarize and Score (SASC) black box text module pre-trained BERT model fMRI voxels language stimuli