Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and
  Accountability
High-quality datasets are fundamental to training and evaluating machine
learning models, yet their creation-especially with accurate human
annotations-remains a significant challenge. Many dataset paper submissions
lack originality, diversity, or rigorous quality control, and these
shortcomings are often overlooked during peer review. Submissions also
frequently omit essential details about dataset construction and properties.
While existing tools such as datasheets aim to promote transparency, they are
largely descriptive and do not provide standardized, measurable methods for
evaluating data quality. Similarly, metadata requirements at conferences
promote accountability but are inconsistently enforced. To address these
limitations, this position paper advocates for the integration of systematic,
rubric-based evaluation metrics into the dataset review process-particularly as
submission volumes continue to grow. We also explore scalable, cost-effective
methods for synthetic data generation, including dedicated tools and
LLM-as-a-judge approaches, to support more efficient evaluation. As a call to
action, we introduce DataRubrics, a structured framework for assessing the
quality of both human- and model-generated datasets. Leveraging recent advances
in LLM-based evaluation, DataRubrics offers a reproducible, scalable, and
actionable solution for dataset quality assessment, enabling both authors and
reviewers to uphold higher standards in data-centric research. We also release
code to support reproducibility of LLM-based evaluations at
https://github.com/datarubrics/datarubrics.
683fb99c7c8d720be438000a
Genta Indra Winata
683fb99c7c8d720be438000b
David Anugraha
683fb99c7c8d720be438000c
Emmy Liu
683fb99c7c8d720be438000d
Alham Fikri Aji
683fb99c7c8d720be438000e
Shou-Yi Hung
683fb99c7c8d720be438000f
Aditya Parashar
683fb99c7c8d720be4380010
Patrick Amadeus Irawan
claimed_verified
2025-06-04T08:55:30.937Z
64d1e3a87e20ec9ea0020d03
https://cdn-avatars.huggingface.co/v1/production/uploads/64d1e3a87e20ec9ea0020d03/xm-afh1AaqS0e9qpaPo7y.jpeg
Patrick Amadeus Irawan
user
patrickamadeus
683fb99c7c8d720be4380011
Ruochen Zhang
683fb99c7c8d720be4380012
Zheng-Xin Yong
683fb99c7c8d720be4380013
Jan Christian Blaise Cruz
claimed_verified
2025-06-16T08:35:43.515Z
5e89aea283fb7a4fc16792bf
https://cdn-avatars.huggingface.co/v1/production/uploads/5e89aea283fb7a4fc16792bf/h_hxnVJdufZHDU8bG-5Db.png
Jan Christian Blaise Cruz
user
jcblaise
683fb99c7c8d720be4380014
Niklas Muennighoff
683fb99c7c8d720be4380015
Seungone Kim
claimed_verified
2025-06-04T08:55:22.158Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
683fb99c7c8d720be4380016
Hanyang Zhao
683fb99c7c8d720be4380017
Sudipta Kar
claimed_verified
2025-06-04T08:55:28.286Z
63139ff6b46fc4e24332fa84
/avatars/ee6923a7cb218f22535064a87761e497.svg
Sudipta Kar
user
cryptexcode
683fb99c7c8d720be4380018
Kezia Erina Suryoraharjo
683fb99c7c8d720be4380019
M. Farid Adilazuarda
683fb99c7c8d720be438001a
En-Shiun Annie Lee
683fb99c7c8d720be438001b
Ayu Purwarianti
683fb99c7c8d720be438001c
Derry Tanti Wijaya
683fb99c7c8d720be438001d
Monojit Choudhury
Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and
  Accountability
High-quality datasets are fundamental to training and evaluating machine
learning models, yet their creation-especially with accurate human
annotations-remains a significant challenge. Many dataset paper submissions
lack originality, diversity, or rigorous quality control, and these
shortcomings are often overlooked during peer review. Submissions also
frequently omit essential details about dataset construction and properties.
While existing tools such as datasheets aim to promote transparency, they are
largely descriptive and do not provide standardized, measurable methods for
evaluating data quality. Similarly, metadata requirements at conferences
promote accountability but are inconsistently enforced. To address these
limitations, this position paper advocates for the integration of systematic,
rubric-based evaluation metrics into the dataset review process-particularly as
submission volumes continue to grow. We also explore scalable, cost-effective
methods for synthetic data generation, including dedicated tools and
LLM-as-a-judge approaches, to support more efficient evaluation. As a call to
action, we introduce DataRubrics, a structured framework for assessing the
quality of both human- and model-generated datasets. Leveraging recent advances
in LLM-based evaluation, DataRubrics offers a reproducible, scalable, and
actionable solution for dataset quality assessment, enabling both authors and
reviewers to uphold higher standards in data-centric research. We also release
code to support reproducibility of LLM-based evaluations at
https://github.com/datarubrics/datarubrics.