Planted in Pretraining, Swayed by Finetuning: A Case Study on the
  Origins of Cognitive Biases in LLMs
Large language models (LLMs) exhibit cognitive biases -- systematic
tendencies of irrational decision-making, similar to those seen in humans.
Prior work has found that these biases vary across models and can be amplified
by instruction tuning. However, it remains unclear if these differences in
biases stem from pretraining, finetuning, or even random noise due to training
stochasticity. We propose a two-step causal experimental approach to
disentangle these factors. First, we finetune models multiple times using
different random seeds to study how training randomness affects over 30
cognitive biases. Second, we introduce cross-tuning -- swapping
instruction datasets between models to isolate bias sources. This swap uses
datasets that led to different bias patterns, directly testing whether biases
are dataset-dependent. Our findings reveal that while training randomness
introduces some variability, biases are mainly shaped by pretraining: models
with the same pretrained backbone exhibit more similar bias patterns than those
sharing only finetuning data. These insights suggest that understanding biases
in finetuned models requires considering their pretraining origins beyond
finetuning effects. This perspective can guide future efforts to develop
principled strategies for evaluating and mitigating bias in LLMs.
68708a2ac8391850d609787d
Itay Itzhak
claimed_verified
2025-07-16T08:15:57.996Z
610c1e1a423fe7d80928aefd
/avatars/8591584d678cf7fddace01e223953a63.svg
Itay Itzhak
user
itay1itzhak
68708a2ac8391850d609787e
Yonatan Belinkov
68708a2ac8391850d609787f
Gabriel Stanovsky
Planted in Pretraining, Swayed by Finetuning: A Case Study on the
  Origins of Cognitive Biases in LLMs
Large language models (LLMs) exhibit cognitive biases -- systematic
tendencies of irrational decision-making, similar to those seen in humans.
Prior work has found that these biases vary across models and can be amplified
by instruction tuning. However, it remains unclear if these differences in
biases stem from pretraining, finetuning, or even random noise due to training
stochasticity. We propose a two-step causal experimental approach to
disentangle these factors. First, we finetune models multiple times using
different random seeds to study how training randomness affects over 30
cognitive biases. Second, we introduce cross-tuning -- swapping
instruction datasets between models to isolate bias sources. This swap uses
datasets that led to different bias patterns, directly testing whether biases
are dataset-dependent. Our findings reveal that while training randomness
introduces some variability, biases are mainly shaped by pretraining: models
with the same pretrained backbone exhibit more similar bias patterns than those
sharing only finetuning data. These insights suggest that understanding biases
in finetuned models requires considering their pretraining origins beyond
finetuning effects. This perspective can guide future efforts to develop
principled strategies for evaluating and mitigating bias in LLMs.
Pretraining significantly shapes cognitive biases in large language models, more so than finetuning or training randomness, as revealed by a causal experimental approach involving cross-tuning.
large language models cognitive biases instruction tuning pretraining finetuning training randomness cross-tuning bias patterns