Language Model Council: Benchmarking Foundation Models on Highly
  Subjective Tasks by Consensus
The rapid advancement of Large Language Models (LLMs) necessitates robust and
challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how
well their responses align with human preferences. However, many tasks such as
those related to emotional intelligence, creative writing, or persuasiveness,
are highly subjective and often lack majoritarian human agreement. Judges may
have irreconcilable disagreements about what constitutes a better response. To
address the challenge of ranking LLMs on highly subjective tasks, we propose a
novel benchmarking framework, the Language Model Council (LMC). The LMC
operates through a democratic process to: 1) formulate a test set through equal
participation, 2) administer the test among council members, and 3) evaluate
responses as a collective jury. We deploy a council of 20 newest LLMs on an
open-ended emotional intelligence task: responding to interpersonal dilemmas.
Our results show that the LMC produces rankings that are more separable,
robust, and less biased than those from any individual LLM judge, and is more
consistent with a human-established leaderboard compared to other benchmarks.
666bcb7b3fc0651814cbfb83
Justin Zhao
extracted_confirmed
2024-06-14T05:07:00.962Z
6462ac71514ee1645bd1f7f7
https://cdn-avatars.huggingface.co/v1/production/uploads/6462ac71514ee1645bd1f7f7/kw2gwI8pyp6uyT4neqV_H.png
Justin Zhao
user
justinxzhao
666bcb7b3fc0651814cbfb84
Flor Miriam Plaza-del-Arco
admin_assigned
2024-06-14T11:28:12.473Z
60f7ec6ad51f9e10a4f422af
/avatars/12ac5037657cfba4e2b48b08c688a629.svg
Flor Miriam Plaza-del-Arco
user
fmplaza
666bcb7b3fc0651814cbfb85
Amanda Cercas Curry
admin_assigned
2024-06-14T11:28:21.671Z
620bacbbc7e4ae30abee681c
/avatars/60710c4627e8924719f3a1c034a6c644.svg
Amanda Curry
user
amandacurry
Language Model Council: Benchmarking Foundation Models on Highly
  Subjective Tasks by Consensus
The rapid advancement of Large Language Models (LLMs) necessitates robust and
challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how
well their responses align with human preferences. However, many tasks such as
those related to emotional intelligence, creative writing, or persuasiveness,
are highly subjective and often lack majoritarian human agreement. Judges may
have irreconcilable disagreements about what constitutes a better response. To
address the challenge of ranking LLMs on highly subjective tasks, we propose a
novel benchmarking framework, the Language Model Council (LMC). The LMC
operates through a democratic process to: 1) formulate a test set through equal
participation, 2) administer the test among council members, and 3) evaluate
responses as a collective jury. We deploy a council of 20 newest LLMs on an
open-ended emotional intelligence task: responding to interpersonal dilemmas.
Our results show that the LMC produces rankings that are more separable,
robust, and less biased than those from any individual LLM judge, and is more
consistent with a human-established leaderboard compared to other benchmarks.
A democratic council framework improves benchmarking of Large Language Models, particularly for subjective tasks like emotional intelligence, by reducing bias and increasing robustness.
Large Language Models Chatbot Arena emotional intelligence creative writing persuasiveness democratic process test set council members collective jury