Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs
  for Text Annotation
Large language models (LLMs) are rapidly transforming social science research
by enabling the automation of labor-intensive tasks like data annotation and
text analysis. However, LLM outputs vary significantly depending on the
implementation choices made by researchers (e.g., model selection, prompting
strategy, or temperature settings). Such variation can introduce systematic
biases and random errors, which propagate to downstream analyses and cause Type
I, Type II, Type S, or Type M errors. We call this LLM hacking.
  We quantify the risk of LLM hacking by replicating 37 data annotation tasks
from 21 published social science research studies with 18 different models.
Analyzing 13 million LLM labels, we test 2,361 realistic hypotheses to measure
how plausible researcher choices affect statistical conclusions. We find
incorrect conclusions based on LLM-annotated data in approximately one in three
hypotheses for state-of-the-art models, and in half the hypotheses for small
language models. While our findings show that higher task performance and
better general model capabilities reduce LLM hacking risk, even highly accurate
models do not completely eliminate it. The risk of LLM hacking decreases as
effect sizes increase, indicating the need for more rigorous verification of
findings near significance thresholds. Our extensive analysis of LLM hacking
mitigation techniques emphasizes the importance of human annotations in
reducing false positive findings and improving model selection. Surprisingly,
common regression estimator correction techniques are largely ineffective in
reducing LLM hacking risk, as they heavily trade off Type I vs. Type II errors.
  Beyond accidental errors, we find that intentional LLM hacking is
unacceptably simple. With few LLMs and just a handful of prompt paraphrases,
anything can be presented as statistically significant.
68c7d6cdee0eed1697d6b82e
Joachim Baumann
claimed_verified
2025-09-15T15:06:14.313Z
6625088168af267420472310
https://cdn-avatars.huggingface.co/v1/production/uploads/6625088168af267420472310/EEAPBs5OXuwJK67dMNi3v.jpeg
Joachim Baumann
user
joebaumann
68c7d6cdee0eed1697d6b82f
Paul Röttger
68c7d6cdee0eed1697d6b830
Aleksandra Urman
68c7d6cdee0eed1697d6b831
Albert Wendsjö
68c7d6cdee0eed1697d6b832
Flor Miriam Plaza-del-Arco
68c7d6cdee0eed1697d6b833
Johannes B. Gruber
68c7d6cdee0eed1697d6b834
Dirk Hovy
Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs
  for Text Annotation
Large language models (LLMs) are rapidly transforming social science research
by enabling the automation of labor-intensive tasks like data annotation and
text analysis. However, LLM outputs vary significantly depending on the
implementation choices made by researchers (e.g., model selection, prompting
strategy, or temperature settings). Such variation can introduce systematic
biases and random errors, which propagate to downstream analyses and cause Type
I, Type II, Type S, or Type M errors. We call this LLM hacking.
  We quantify the risk of LLM hacking by replicating 37 data annotation tasks
from 21 published social science research studies with 18 different models.
Analyzing 13 million LLM labels, we test 2,361 realistic hypotheses to measure
how plausible researcher choices affect statistical conclusions. We find
incorrect conclusions based on LLM-annotated data in approximately one in three
hypotheses for state-of-the-art models, and in half the hypotheses for small
language models. While our findings show that higher task performance and
better general model capabilities reduce LLM hacking risk, even highly accurate
models do not completely eliminate it. The risk of LLM hacking decreases as
effect sizes increase, indicating the need for more rigorous verification of
findings near significance thresholds. Our extensive analysis of LLM hacking
mitigation techniques emphasizes the importance of human annotations in
reducing false positive findings and improving model selection. Surprisingly,
common regression estimator correction techniques are largely ineffective in
reducing LLM hacking risk, as they heavily trade off Type I vs. Type II errors.
  Beyond accidental errors, we find that intentional LLM hacking is
unacceptably simple. With few LLMs and just a handful of prompt paraphrases,
anything can be presented as statistically significant.
LLM hacking introduces significant variability and error in social science research, affecting statistical conclusions and requiring rigorous verification and human annotations to mitigate.
large language models LLMs data annotation text analysis model selection prompting strategy temperature settings systematic biases random errors Type I errors Type II errors Type S errors Type M errors LLM hacking statistical conclusions effect sizes human annotations regression estimator correction