BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model
  Responses
Existing studies on bias mitigation methods for large language models (LLMs)
use diverse baselines and metrics to evaluate debiasing performance, leading to
inconsistent comparisons among them. Moreover, their evaluations are mostly
based on the comparison between LLMs' probabilities of biased and unbiased
contexts, which ignores the gap between such evaluations and real-world use
cases where users interact with LLMs by reading model responses and expect fair
and safe outputs rather than LLMs' probabilities. To enable consistent
evaluation across debiasing methods and bridge this gap, we introduce
BiasFreeBench, an empirical benchmark that comprehensively compares eight
mainstream bias mitigation techniques (covering four prompting-based and four
training-based methods) on two test scenarios (multi-choice QA and open-ended
multi-turn QA) by reorganizing existing datasets into a unified query-response
setting. We further introduce a response-level metric, Bias-Free Score, to
measure the extent to which LLM responses are fair, safe, and
anti-stereotypical. Debiasing performances are systematically compared and
analyzed across key dimensions: the prompting vs. training paradigm, model
size, and generalization of different training strategies to unseen bias types.
We will publicly release our benchmark, aiming to establish a unified testbed
for bias mitigation research.
68ddd96f6024653e8a3ed06d
Xin Xu
claimed_verified
2025-10-02T03:07:58.427Z
6190ab805ca89a28e9f66873
/avatars/3c7ecc398fbf851acd2a132e947a92be.svg
Xin Xu
user
XinXuNLPer
68ddd96f6024653e8a3ed06e
Xunzhi He
68ddd96f6024653e8a3ed06f
Churan Zhi
68ddd96f6024653e8a3ed070
Ruizhe Chen
68ddd96f6024653e8a3ed071
Julian McAuley
68ddd96f6024653e8a3ed072
Zexue He
BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model
  Responses
Existing studies on bias mitigation methods for large language models (LLMs)
use diverse baselines and metrics to evaluate debiasing performance, leading to
inconsistent comparisons among them. Moreover, their evaluations are mostly
based on the comparison between LLMs' probabilities of biased and unbiased
contexts, which ignores the gap between such evaluations and real-world use
cases where users interact with LLMs by reading model responses and expect fair
and safe outputs rather than LLMs' probabilities. To enable consistent
evaluation across debiasing methods and bridge this gap, we introduce
BiasFreeBench, an empirical benchmark that comprehensively compares eight
mainstream bias mitigation techniques (covering four prompting-based and four
training-based methods) on two test scenarios (multi-choice QA and open-ended
multi-turn QA) by reorganizing existing datasets into a unified query-response
setting. We further introduce a response-level metric, Bias-Free Score, to
measure the extent to which LLM responses are fair, safe, and
anti-stereotypical. Debiasing performances are systematically compared and
analyzed across key dimensions: the prompting vs. training paradigm, model
size, and generalization of different training strategies to unseen bias types.
We will publicly release our benchmark, aiming to establish a unified testbed
for bias mitigation research.
BiasFreeBench evaluates bias mitigation techniques in large language models using a unified benchmark and response-level metric to ensure fair and safe outputs in real-world scenarios.
large language models bias mitigation BiasFreeBench prompting-based methods training-based methods multi-choice QA open-ended multi-turn QA Bias-Free Score model size generalization