B-score: Detecting biases in large language models using response
  history
Large language models (LLMs) often exhibit strong biases, e.g, against women
or in favor of the number 7. We investigate whether LLMs would be able to
output less biased answers when allowed to observe their prior answers to the
same question in a multi-turn conversation. To understand which types of
questions invite more biased answers, we test LLMs on our proposed set of
questions that span 9 topics and belong to three types: (1) Subjective; (2)
Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves
in a multi-turn conversation in response to questions that seek an Random,
unbiased answer. Furthermore, we propose B-score, a novel metric that is
effective in detecting biases to Subjective, Random, Easy, and Hard questions.
On MMLU, HLE, and CSQA, leveraging B-score substantially improves the
verification accuracy of LLM answers (i.e, accepting LLM correct answers and
rejecting incorrect ones) compared to using verbalized confidence scores or the
frequency of single-turn answers alone. Code and data are available at:
https://b-score.github.io.
6835217ee759f596d018f72c
An Vo
claimed_verified
2025-05-27T07:52:08.994Z
6631fd5961a4305e5610d403
https://cdn-avatars.huggingface.co/v1/production/uploads/6631fd5961a4305e5610d403/P1Dtxzn-KIbYDDsiw60nr.jpeg
An Vo
user
anvo25
6835217ee759f596d018f72d
Mohammad Reza Taesiri
extracted_confirmed
2025-05-27T02:21:46.082Z
6039478ab3ecf716b1a5fd4d
https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg
taesiri
user
taesiri
6835217ee759f596d018f72e
Daeyoung Kim
6835217ee759f596d018f72f
Anh Totti Nguyen
extracted_confirmed
2025-06-20T19:47:53.519Z
60e85b3fcd1cf4e418fff651
https://cdn-avatars.huggingface.co/v1/production/uploads/60e85b3fcd1cf4e418fff651/TUaEA4SqB6xuG2JhwNETC.jpeg
Anh (Totti) Nguyen
user
anhng8
B-score: Detecting biases in large language models using response
  history
Large language models (LLMs) often exhibit strong biases, e.g, against women
or in favor of the number 7. We investigate whether LLMs would be able to
output less biased answers when allowed to observe their prior answers to the
same question in a multi-turn conversation. To understand which types of
questions invite more biased answers, we test LLMs on our proposed set of
questions that span 9 topics and belong to three types: (1) Subjective; (2)
Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves
in a multi-turn conversation in response to questions that seek an Random,
unbiased answer. Furthermore, we propose B-score, a novel metric that is
effective in detecting biases to Subjective, Random, Easy, and Hard questions.
On MMLU, HLE, and CSQA, leveraging B-score substantially improves the
verification accuracy of LLM answers (i.e, accepting LLM correct answers and
rejecting incorrect ones) compared to using verbalized confidence scores or the
frequency of single-turn answers alone. Code and data are available at:
https://b-score.github.io.
LLMs can reduce biases in multi-turn conversations for certain types of questions, and a novel B-score metric improves the accuracy of verifying LLM answers.
large language models biases multi-turn conversation B-score MMLU HLE CSQA verification accuracy verbalized confidence scores