Atla Selene Mini: A General Purpose Evaluation Model
We introduce Atla Selene Mini, a state-of-the-art small language
model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that
outperforms the best SLMJs and GPT-4o-mini on overall performance across 11
out-of-distribution benchmarks, spanning absolute scoring, classification, and
pairwise preference tasks. It is the highest-scoring 8B generative model on
RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To
achieve this, we develop a principled data curation strategy that augments
public datasets with synthetically generated critiques and ensures high quality
through filtering and dataset ablations. We train our model on a combined
direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and
produce a highly promptable evaluator that excels in real-world scenarios.
Selene Mini shows dramatically improved zero-shot agreement with human expert
evaluations on financial and medical industry datasets. It is also robust to
variations in prompt format. Preliminary results indicate that Selene Mini is
the top-ranking evaluator in a live, community-driven Judge Arena. We release
the model weights on HuggingFace
(https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage
widespread community adoption.
679ae7655c55250b48483742
Andrei Alexandru
claimed_verified
2025-01-30T15:17:07.941Z
62571e9e0e0c97db812e3afb
https://cdn-avatars.huggingface.co/v1/production/uploads/1662586587273-62571e9e0e0c97db812e3afb.jpeg
Andrei Alexandru
user
inwaves
679ae7655c55250b48483743
Antonia Calvi
admin_assigned
2025-01-30T09:40:54.827Z
66e184e86048d62cd8fb4e52
/avatars/bc633262648dd5335171e5372552d67d.svg
Antonia Calvi
user
NinaCalvi
679ae7655c55250b48483744
Henry Broomfield
679ae7655c55250b48483745
Jackson Golden
679ae7655c55250b48483746
Kyle Dai
claimed_verified
2025-01-30T15:17:03.347Z
659fc8832cb13cede03047bb
https://cdn-avatars.huggingface.co/v1/production/uploads/659fc8832cb13cede03047bb/Wo_LjryGEJFnxXrOcokfE.jpeg
kyle
user
kaikaidai
679ae7655c55250b48483747
Mathias Leys
679ae7655c55250b48483748
Maurice Burger
claimed_verified
2025-01-30T15:17:05.147Z
66d08d5c952f5e4e64bd6be0
/avatars/2fc3a6e3813718f0c001fb26337dab45.svg
Maurice
user
MauriceBurg
679ae7655c55250b48483749
Max Bartolo
679ae7655c55250b4848374a
Roman Engeler
679ae7655c55250b4848374b
Sashank Pisupati
claimed_verified
2025-01-30T15:17:06.579Z
633c4fb100732349209f2aad
/avatars/b44ccae4fb097284730291e4fcc47a24.svg
Sashank Pisupati
user
spisupat
679ae7655c55250b4848374c
Toby Drane
679ae7655c55250b4848374d
Young Sun Park
Atla Selene Mini: A General Purpose Evaluation Model
We introduce Atla Selene Mini, a state-of-the-art small language
model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that
outperforms the best SLMJs and GPT-4o-mini on overall performance across 11
out-of-distribution benchmarks, spanning absolute scoring, classification, and
pairwise preference tasks. It is the highest-scoring 8B generative model on
RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To
achieve this, we develop a principled data curation strategy that augments
public datasets with synthetically generated critiques and ensures high quality
through filtering and dataset ablations. We train our model on a combined
direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and
produce a highly promptable evaluator that excels in real-world scenarios.
Selene Mini shows dramatically improved zero-shot agreement with human expert
evaluations on financial and medical industry datasets. It is also robust to
variations in prompt format. Preliminary results indicate that Selene Mini is
the top-ranking evaluator in a live, community-driven Judge Arena. We release
the model weights on HuggingFace
(https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage
widespread community adoption.
Atla Selene Mini, an 8B language model-as-a-judge, excels across various benchmarks using enhanced data curation and a combined training approach, achieving top performance in zero-shot evaluations and real-world scenarios.
data curation synthetically generated critiques direct preference optimization supervised fine-tuning promptable evaluator zero-shot agreement human expert evaluations