YourBench: Easy Custom Evaluation Sets for Everyone
Evaluating large language models (LLMs) effectively remains a critical
bottleneck, as traditional static benchmarks suffer from saturation and
contamination, while human evaluations are costly and slow. This hinders timely
or domain-specific assessment, crucial for real-world applications. We
introduce YourBench, a novel, open-source framework that addresses these
limitations by enabling dynamic, automated generation of reliable, up-to-date,
and domain-tailored benchmarks cheaply and without manual annotation, directly
from user-provided documents. We demonstrate its efficacy by replicating 7
diverse MMLU subsets using minimal source text, achieving this for under 15 USD
in total inference costs while perfectly preserving the relative model
performance rankings (Spearman Rho = 1) observed on the original benchmark. To
ensure that YourBench generates data grounded in provided input instead of
relying on posterior parametric knowledge in models, we also introduce
Tempora-0325, a novel dataset of over 7K diverse documents, published
exclusively after March 2025. Our comprehensive analysis spans 26 SoTA models
from 7 major families across varying scales (3-671B parameters) to validate the
quality of generated evaluations through rigorous algorithmic checks (e.g.,
citation grounding) and human assessments. We release the YourBench library,
the Tempora-0325 dataset, 150k+ question answer pairs based on Tempora and all
evaluation and inference traces to facilitate reproducible research and empower
the community to generate bespoke benchmarks on demand, fostering more relevant
and trustworthy LLM evaluation.
67eddf485224e5e2c1a39491
Sumuk Shashidhar
extracted_confirmed
2025-05-31T02:59:18.819Z
64676c81e7a6a374fd181110
https://cdn-avatars.huggingface.co/v1/production/uploads/64676c81e7a6a374fd181110/HEDlAT9FYhJF9Nw__LLNI.jpeg
Sumuk Shashidhar
user
sumuks
67eddf485224e5e2c1a39492
Clémentine Fourrier
extracted_pending
2025-04-03T01:07:21.792Z
6202a599216215a22221dea9
https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
Clémentine Fourrier
user
clefourrier
67eddf485224e5e2c1a39493
Alina Lozovskia
67eddf485224e5e2c1a39494
Thomas Wolf
claimed_verified
2025-04-08T06:57:10.557Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
67eddf485224e5e2c1a39495
Gokhan Tur
67eddf485224e5e2c1a39496
Dilek Hakkani-Tür
claimed_verified
2025-06-16T15:51:20.344Z
6747519282137543033451d9
/avatars/8bd763f19d8e9f97f0f6817435b062dc.svg
Dilek Hakkani-Tur
user
dilekht
YourBench: Easy Custom Evaluation Sets for Everyone
Evaluating large language models (LLMs) effectively remains a critical
bottleneck, as traditional static benchmarks suffer from saturation and
contamination, while human evaluations are costly and slow. This hinders timely
or domain-specific assessment, crucial for real-world applications. We
introduce YourBench, a novel, open-source framework that addresses these
limitations by enabling dynamic, automated generation of reliable, up-to-date,
and domain-tailored benchmarks cheaply and without manual annotation, directly
from user-provided documents. We demonstrate its efficacy by replicating 7
diverse MMLU subsets using minimal source text, achieving this for under 15 USD
in total inference costs while perfectly preserving the relative model
performance rankings (Spearman Rho = 1) observed on the original benchmark. To
ensure that YourBench generates data grounded in provided input instead of
relying on posterior parametric knowledge in models, we also introduce
Tempora-0325, a novel dataset of over 7K diverse documents, published
exclusively after March 2025. Our comprehensive analysis spans 26 SoTA models
from 7 major families across varying scales (3-671B parameters) to validate the
quality of generated evaluations through rigorous algorithmic checks (e.g.,
citation grounding) and human assessments. We release the YourBench library,
the Tempora-0325 dataset, 150k+ question answer pairs based on Tempora and all
evaluation and inference traces to facilitate reproducible research and empower
the community to generate bespoke benchmarks on demand, fostering more relevant
and trustworthy LLM evaluation.
YourBench introduces an automated framework for generating domain-specific LLM benchmarks dynamically and cost-effectively from user documents, ensuring reliable and up-to-date evaluations.
large language models YourBench dynamic benchmarks automated generation domain-tailored benchmarks Tempora-0325 MMLU subsets Tempora-0325 dataset citation grounding SoTA models parameter-efficient fine-tuning reproducible research