I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating
  Linguistic Shibboleth Detection in LLM Hiring Evaluations
This paper introduces a comprehensive benchmark for evaluating how Large
Language Models (LLMs) respond to linguistic shibboleths: subtle linguistic
markers that can inadvertently reveal demographic attributes such as gender,
social class, or regional background. Through carefully constructed interview
simulations using 100 validated question-response pairs, we demonstrate how
LLMs systematically penalize certain linguistic patterns, particularly hedging
language, despite equivalent content quality. Our benchmark generates
controlled linguistic variations that isolate specific phenomena while
maintaining semantic equivalence, which enables the precise measurement of
demographic bias in automated evaluation systems. We validate our approach
along multiple linguistic dimensions, showing that hedged responses receive
25.6% lower ratings on average, and demonstrate the benchmark's effectiveness
in identifying model-specific biases. This work establishes a foundational
framework for detecting and measuring linguistic discrimination in AI systems,
with broad applications to fairness in automated decision-making contexts.
6895813348b0ae5ca2710e03
Julia Kharchenko
6895813348b0ae5ca2710e04
Tanya Roosta
6895813348b0ae5ca2710e05
Aman Chadha
claimed_verified
2025-08-08T16:19:09.855Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
6895813348b0ae5ca2710e06
Chirag Shah
I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating
  Linguistic Shibboleth Detection in LLM Hiring Evaluations
This paper introduces a comprehensive benchmark for evaluating how Large
Language Models (LLMs) respond to linguistic shibboleths: subtle linguistic
markers that can inadvertently reveal demographic attributes such as gender,
social class, or regional background. Through carefully constructed interview
simulations using 100 validated question-response pairs, we demonstrate how
LLMs systematically penalize certain linguistic patterns, particularly hedging
language, despite equivalent content quality. Our benchmark generates
controlled linguistic variations that isolate specific phenomena while
maintaining semantic equivalence, which enables the precise measurement of
demographic bias in automated evaluation systems. We validate our approach
along multiple linguistic dimensions, showing that hedged responses receive
25.6% lower ratings on average, and demonstrate the benchmark's effectiveness
in identifying model-specific biases. This work establishes a foundational
framework for detecting and measuring linguistic discrimination in AI systems,
with broad applications to fairness in automated decision-making contexts.
A benchmark evaluates Large Language Models' response to linguistic markers that reveal demographic attributes, demonstrating systematic penalization of hedging language despite equivalent content quality.
Large Language Models linguistic shibboleths hedging language interview simulations linguistic dimensions demographic bias automated evaluation systems linguistic discrimination fairness in automated decision-making