Counterfactual Generation from Language Models
Understanding and manipulating the causal generation mechanisms in language
models is essential for controlling their behavior. Previous work has primarily
relied on techniques such as representation surgery -- e.g., model ablations or
manipulation of linear subspaces tied to specific concepts -- to intervene on
these models. To understand the impact of interventions precisely, it is useful
to examine counterfactuals -- e.g., how a given sentence would have appeared
had it been generated by the model following a specific intervention. We
highlight that counterfactual reasoning is conceptually distinct from
interventions, as articulated in Pearl's causal hierarchy. Based on this
observation, we propose a framework for generating true string counterfactuals
by reformulating language models as Generalized Structural-equation. Models
using the Gumbel-max trick. This allows us to model the joint distribution over
original strings and their counterfactuals resulting from the same
instantiation of the sampling noise. We develop an algorithm based on hindsight
Gumbel sampling that allows us to infer the latent noise variables and generate
counterfactuals of observed strings. Our experiments demonstrate that the
approach produces meaningful counterfactuals while at the same time showing
that commonly used intervention techniques have considerable undesired side
effects.
6732cc097adce85d6b98eda1
Shauli Ravfogel
claimed_verified
2024-11-15T09:27:04.233Z
635686ec5aeb69011c7d1abd
/avatars/c59034ad2c9c2daf4b4a8d3c56449f5e.svg
Shauli Ravfogel
user
ravfogs
6732cc097adce85d6b98eda2
Anej Svete
claimed_verified
2024-11-21T21:33:12.209Z
6629228e0e7290809cd6b243
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/6KfT9MexmALsAywliij5V.jpeg
Anej Svete
user
AnejSvete
6732cc097adce85d6b98eda3
Vésteinn Snæbjarnarson
extracted_confirmed
2025-07-28T02:26:40.795Z
5f915e09161dc51925934edf
https://cdn-avatars.huggingface.co/v1/production/uploads/5f915e09161dc51925934edf/2PqjbJ7FVG3FOCNmn7ORY.png
Vésteinn Snæbjarnarson
user
vesteinn
6732cc097adce85d6b98eda4
Ryan Cotterell
Counterfactual Generation from Language Models
Understanding and manipulating the causal generation mechanisms in language
models is essential for controlling their behavior. Previous work has primarily
relied on techniques such as representation surgery -- e.g., model ablations or
manipulation of linear subspaces tied to specific concepts -- to intervene on
these models. To understand the impact of interventions precisely, it is useful
to examine counterfactuals -- e.g., how a given sentence would have appeared
had it been generated by the model following a specific intervention. We
highlight that counterfactual reasoning is conceptually distinct from
interventions, as articulated in Pearl's causal hierarchy. Based on this
observation, we propose a framework for generating true string counterfactuals
by reformulating language models as Generalized Structural-equation. Models
using the Gumbel-max trick. This allows us to model the joint distribution over
original strings and their counterfactuals resulting from the same
instantiation of the sampling noise. We develop an algorithm based on hindsight
Gumbel sampling that allows us to infer the latent noise variables and generate
counterfactuals of observed strings. Our experiments demonstrate that the
approach produces meaningful counterfactuals while at the same time showing
that commonly used intervention techniques have considerable undesired side
effects.
A framework using Generalized Structural Equation Models and Gumbel-max trick allows generation of meaningful counterfactuals in language models, revealing side effects of common intervention techniques.
language models representation surgery counterfactuals Pearl's causal hierarchy Generalized Structural-equation Models Gumbel-max trick hindsight Gumbel sampling latent noise variables