DISCO: Diversifying Sample Condensation for Efficient Model Evaluation
Evaluating modern machine learning models has become prohibitively expensive.
Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model.
Costly evaluation reduces inclusivity, slows the cycle of innovation, and
worsens environmental impact. The typical approach follows two steps. First,
select an anchor subset of data. Second, train a mapping from the accuracy on
this subset to the final test result. The drawback is that anchor selection
depends on clustering, which can be complex and sensitive to design choices. We
argue that promoting diversity among samples is not essential; what matters is
to select samples that maximise diversity in model responses. Our
method, Diversifying Sample Condensation (DISCO), selects the top-k
samples with the greatest model disagreements. This uses greedy, sample-wise
statistics rather than global clustering. The approach is conceptually simpler.
From a theoretical view, inter-model disagreement provides an
information-theoretically optimal rule for such greedy selection.
DISCO shows empirical gains over prior methods, achieving
state-of-the-art results in performance prediction across MMLU, Hellaswag,
Winogrande, and ARC. Code is available here:
https://github.com/arubique/disco-public.
68eca176cd07fb414898cb10
Alexander Rubinstein
68eca176cd07fb414898cb11
Benjamin Raible
68eca176cd07fb414898cb12
Martin Gubri
claimed_verified
2025-10-13T10:04:20.876Z
63ee35c3f599efc7a010c792
https://cdn-avatars.huggingface.co/v1/production/uploads/1676555600618-noauth.jpeg
Martin Gubri
user
mgubri
68eca176cd07fb414898cb13
Seong Joon Oh
DISCO: Diversifying Sample Condensation for Efficient Model Evaluation
Evaluating modern machine learning models has become prohibitively expensive.
Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model.
Costly evaluation reduces inclusivity, slows the cycle of innovation, and
worsens environmental impact. The typical approach follows two steps. First,
select an anchor subset of data. Second, train a mapping from the accuracy on
this subset to the final test result. The drawback is that anchor selection
depends on clustering, which can be complex and sensitive to design choices. We
argue that promoting diversity among samples is not essential; what matters is
to select samples that maximise diversity in model responses. Our
method, Diversifying Sample Condensation (DISCO), selects the top-k
samples with the greatest model disagreements. This uses greedy, sample-wise
statistics rather than global clustering. The approach is conceptually simpler.
From a theoretical view, inter-model disagreement provides an
information-theoretically optimal rule for such greedy selection.
DISCO shows empirical gains over prior methods, achieving
state-of-the-art results in performance prediction across MMLU, Hellaswag,
Winogrande, and ARC. Code is available here:
https://github.com/arubique/disco-public.
A method called DISCO selects samples with the greatest model disagreements to predict performance, achieving state-of-the-art results across various benchmarks with reduced computational cost.
Diversifying Sample Condensation (DISCO) model disagreements greedy selection information-theoretically optimal rule MMLU Hellaswag Winogrande ARC