Position: AI Competitions Provide the Gold Standard for Empirical Rigor
  in GenAI Evaluation
In this position paper, we observe that empirical evaluation in Generative AI
is at a crisis point since traditional ML evaluation and benchmarking
strategies are insufficient to meet the needs of evaluating modern GenAI models
and systems. There are many reasons for this, including the fact that these
models typically have nearly unbounded input and output spaces, typically do
not have a well defined ground truth target, and typically exhibit strong
feedback loops and prediction dependence based on context of previous model
outputs. On top of these critical issues, we argue that the problems of {\em
leakage} and {\em contamination} are in fact the most important and difficult
issues to address for GenAI evaluations. Interestingly, the field of AI
Competitions has developed effective measures and practices to combat leakage
for the purpose of counteracting cheating by bad actors within a competition
setting. This makes AI Competitions an especially valuable (but underutilized)
resource. Now is time for the field to view AI Competitions as the gold
standard for empirical rigor in GenAI evaluation, and to harness and harvest
their results with according value.
68233e50c28353126f22bf74
D. Sculley
68233e50c28353126f22bf75
Will Cukierski
68233e50c28353126f22bf76
Phil Culliton
claimed_verified
2025-05-14T07:35:34.570Z
627a9aa0f967ef475abe8037
/avatars/47cf94d6c015577d0a860b617c9ea2b8.svg
Phil Culliton
user
PhilCulliton
68233e50c28353126f22bf77
Sohier Dane
claimed_verified
2025-05-14T07:35:40.368Z
6823721744c921f21ffc23dd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BZQBui3e4JMVGUzBgQBC3.png
Sohier Dane
user
SohierDane
68233e50c28353126f22bf78
Maggie Demkin
68233e50c28353126f22bf79
Ryan Holbrook
68233e50c28353126f22bf7a
Addison Howard
claimed_verified
2025-05-14T07:35:49.163Z
68234703b30c1a41e2ea1387
/avatars/5842f616f01f970696227d6eeb83a30b.svg
Addison Howard
user
ahoward14
68233e50c28353126f22bf7b
Paul Mooney
claimed_verified
2025-05-14T07:35:37.635Z
627a89f393d0b645835e57aa
https://cdn-avatars.huggingface.co/v1/production/uploads/1652197873133-noauth.jpeg
Paul M
user
paultimothymooney
68233e50c28353126f22bf7c
Walter Reade
claimed_verified
2025-05-14T07:35:46.514Z
63bf3b2287619d1458cddbe0
https://cdn-avatars.huggingface.co/v1/production/uploads/1673476847649-noauth.jpeg
Walter Reade
user
walterreade
68233e50c28353126f22bf7d
Megan Risdal
extracted_pending
2025-05-13T13:17:20.366Z
626b23fc822e3d85324babed
https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
Meg Risdal
user
megrisdal
68233e50c28353126f22bf7e
Nate Keating
claimed_verified
2025-05-14T07:35:43.363Z
621656e913bc8271a06798a0
https://cdn-avatars.huggingface.co/v1/production/uploads/621656e913bc8271a06798a0/2cmQEKQG3fuhKy45WLPoQ.jpeg
Nate Keating
user
natekeating
Position: AI Competitions Provide the Gold Standard for Empirical Rigor
  in GenAI Evaluation
In this position paper, we observe that empirical evaluation in Generative AI
is at a crisis point since traditional ML evaluation and benchmarking
strategies are insufficient to meet the needs of evaluating modern GenAI models
and systems. There are many reasons for this, including the fact that these
models typically have nearly unbounded input and output spaces, typically do
not have a well defined ground truth target, and typically exhibit strong
feedback loops and prediction dependence based on context of previous model
outputs. On top of these critical issues, we argue that the problems of {\em
leakage} and {\em contamination} are in fact the most important and difficult
issues to address for GenAI evaluations. Interestingly, the field of AI
Competitions has developed effective measures and practices to combat leakage
for the purpose of counteracting cheating by bad actors within a competition
setting. This makes AI Competitions an especially valuable (but underutilized)
resource. Now is time for the field to view AI Competitions as the gold
standard for empirical rigor in GenAI evaluation, and to harness and harvest
their results with according value.
Empirical evaluation of Generative AI models faces significant challenges due to unbounded input/output spaces, lack of well-defined ground truth, feedback loops, and issues like leakage and contamination, suggesting AI Competitions as a gold standard for robust evaluation.
Generative AI empirical evaluation ML evaluation benchmarking input/output spaces ground truth feedback loops prediction dependence leakage contamination AI Competitions