Reliable, Reproducible, and Really Fast Leaderboards with Evalica
The rapid advancement of natural language processing (NLP) technologies, such
as instruction-tuned large language models (LLMs), urges the development of
modern evaluation protocols with human and machine feedback. We introduce
Evalica, an open-source toolkit that facilitates the creation of reliable and
reproducible model leaderboards. This paper presents its design, evaluates its
performance, and demonstrates its usability through its Web interface,
command-line interface, and Python API.
67613d66896c29e2a598e378
Dmitry Ustalov
extracted_confirmed
2024-12-17T08:59:57.545Z
60ef34ff8432bc401cd0abbf
/avatars/4992b27a8230b80e9bb9a4f92bd5638c.svg
Dmitry Ustalov
user
dustalov
Reliable, Reproducible, and Really Fast Leaderboards with Evalica
The rapid advancement of natural language processing (NLP) technologies, such
as instruction-tuned large language models (LLMs), urges the development of
modern evaluation protocols with human and machine feedback. We introduce
Evalica, an open-source toolkit that facilitates the creation of reliable and
reproducible model leaderboards. This paper presents its design, evaluates its
performance, and demonstrates its usability through its Web interface,
command-line interface, and Python API.
Evalica is an open-source toolkit designed to create reliable and reproducible leaderboards for instruction-tuned large language models with support for human and machine feedback.
instruction-tuned large language models Evalica model leaderboards Web interface command-line interface Python API