The Generative Energy Arena (GEA): Incorporating Energy Awareness in
  Large Language Model (LLM) Human Evaluations
The evaluation of large language models is a complex task, in which several
approaches have been proposed. The most common is the use of automated
benchmarks in which LLMs have to answer multiple-choice questions of different
topics. However, this method has certain limitations, being the most
concerning, the poor correlation with the humans. An alternative approach, is
to have humans evaluate the LLMs. This poses scalability issues as there is a
large and growing number of models to evaluate making it impractical (and
costly) to run traditional studies based on recruiting a number of evaluators
and having them rank the responses of the models. An alternative approach is
the use of public arenas, such as the popular LM arena, on which any user can
freely evaluate models on any question and rank the responses of two models.
The results are then elaborated into a model ranking. An increasingly important
aspect of LLMs is their energy consumption and, therefore, evaluating how
energy awareness influences the decisions of humans in selecting a model is of
interest. In this paper, we present GEA, the Generative Energy Arena, an arena
that incorporates information on the energy consumption of the model in the
evaluation process. Preliminary results obtained with GEA are also presented,
showing that for most questions, when users are aware of the energy
consumption, they favor smaller and more energy efficient models. This suggests
that for most user interactions, the extra cost and energy incurred by the more
complex and top-performing models do not provide an increase in the perceived
quality of the responses that justifies their use.
687df30d2e8db0930be6f168
Carlos Arriaga
687df30d2e8db0930be6f169
Gonzalo Martínez
admin_assigned
2025-07-21T16:04:55.307Z
64f31365ed48e3bb9c487d5d
/avatars/979c1979eadbd4529c95b925bbb58d78.svg
Gonzalo
user
gonzmart
687df30d2e8db0930be6f16a
Eneko Sendin
admin_assigned
2025-07-21T16:05:15.390Z
67a72041e113700e5e931699
/avatars/383882ca36f60ca89c311d3370a4413e.svg
Eneko Sendín
user
EESG
687df30d2e8db0930be6f16b
Javier Conde
admin_assigned
2025-07-21T16:05:08.768Z
66852115f8c3f32152346733
https://cdn-avatars.huggingface.co/v1/production/uploads/66852115f8c3f32152346733/EuApIooUvMesCSr27Auze.jpeg
Javier Conde
user
javicond3
687df30d2e8db0930be6f16c
Pedro Reviriego
admin_assigned
2025-07-21T16:05:01.701Z
6574f66b06fdcd4ca9491299
/avatars/e31821949d75efb750ab2d9ebe12b9a8.svg
pedro reviriego
user
reviriego
The Generative Energy Arena (GEA): Incorporating Energy Awareness in
  Large Language Model (LLM) Human Evaluations
The evaluation of large language models is a complex task, in which several
approaches have been proposed. The most common is the use of automated
benchmarks in which LLMs have to answer multiple-choice questions of different
topics. However, this method has certain limitations, being the most
concerning, the poor correlation with the humans. An alternative approach, is
to have humans evaluate the LLMs. This poses scalability issues as there is a
large and growing number of models to evaluate making it impractical (and
costly) to run traditional studies based on recruiting a number of evaluators
and having them rank the responses of the models. An alternative approach is
the use of public arenas, such as the popular LM arena, on which any user can
freely evaluate models on any question and rank the responses of two models.
The results are then elaborated into a model ranking. An increasingly important
aspect of LLMs is their energy consumption and, therefore, evaluating how
energy awareness influences the decisions of humans in selecting a model is of
interest. In this paper, we present GEA, the Generative Energy Arena, an arena
that incorporates information on the energy consumption of the model in the
evaluation process. Preliminary results obtained with GEA are also presented,
showing that for most questions, when users are aware of the energy
consumption, they favor smaller and more energy efficient models. This suggests
that for most user interactions, the extra cost and energy incurred by the more
complex and top-performing models do not provide an increase in the perceived
quality of the responses that justifies their use.
GEA, a public arena that incorporates energy consumption data, shows that users often prefer smaller, more energy-efficient language models over larger, more complex ones.
large language models automated benchmarks multiple-choice questions human evaluation scalability issues public arenas LM arena energy consumption energy awareness model ranking generative energy arena GEA