MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
Perceiving and generating diverse modalities are crucial for AI models to
effectively learn from and engage with real-world signals, necessitating
reliable evaluations for their development. We identify two major issues in
current evaluations: (1) inconsistent standards, shaped by different
communities with varying protocols and maturity levels; and (2) significant
query, grading, and generalization biases. To address these, we introduce
MixEval-X, the first any-to-any real-world benchmark designed to optimize and
standardize evaluations across input and output modalities. We propose
multi-modal benchmark mixture and adaptation-rectification pipelines to
reconstruct real-world task distributions, ensuring evaluations generalize
effectively to real-world use cases. Extensive meta-evaluations show our
approach effectively aligns benchmark samples with real-world task
distributions and the model rankings correlate strongly with that of
crowd-sourced real-world evaluations (up to 0.98). We provide comprehensive
leaderboards to rerank existing models and organizations and offer insights to
enhance understanding of multi-modal evaluations and inform future research.
6711de895b7360457e26dff7
Jinjie Ni
claimed_verified
2024-10-18T08:55:08.722Z
62a7362fd1e7a011fd4e31a7
https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
Jinjie Ni
user
jinjieni
6711de895b7360457e26dff8
Yifan Song
6711de895b7360457e26dff9
Deepanway Ghosal
admin_assigned
2024-10-18T09:31:24.961Z
62eced5e7e89e8d34df1a1ea
/avatars/99c9d8ba7e7722b7524d5d687cf96a25.svg
Deepanway Ghosal
user
dghosal
6711de895b7360457e26dffa
Bo Li
claimed_verified
2024-10-21T07:48:14.198Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
6711de895b7360457e26dffb
David Junhao Zhang
admin_assigned
2024-10-18T10:33:07.038Z
6468855763a564ba347e1efa
/avatars/caa99c3318c1d6d21be44d2d32795a62.svg
David Junhao ZHANG
user
Junhao233
6711de895b7360457e26dffc
Xiang Yue
claimed_verified
2024-10-18T14:55:42.911Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
6711de895b7360457e26dffd
Fuzhao Xue
admin_assigned
2024-10-18T10:33:15.728Z
64a52314978242bb77e05790
/avatars/e97206c653de3446266c80c2943c0a6f.svg
Fuzhao Xue
user
fuzhao
6711de895b7360457e26dffe
Zian Zheng
claimed_verified
2024-10-21T07:48:16.263Z
6371e32aafbe42caa5ad9189
https://cdn-avatars.huggingface.co/v1/production/uploads/6371e32aafbe42caa5ad9189/Frqjr_L1BGyg4yWwVXKiv.jpeg
Zheng Zian(Andy)
user
OrionZheng
6711de895b7360457e26dfff
Kaichen Zhang
claimed_verified
2024-10-18T08:12:13.789Z
64bb77e786e7fb5b8a317a43
https://cdn-avatars.huggingface.co/v1/production/uploads/64bb77e786e7fb5b8a317a43/J0jOrlZJ9gazdYaeSH2Bo.png
kcz
user
kcz358
6711de895b7360457e26e000
Mahir Shah
6711de895b7360457e26e001
Kabir Jain
claimed_verified
2024-10-21T07:48:17.775Z
648763974c7531fdad63de41
/avatars/ca629fa7aa0749d85c1e3fa3f44f009e.svg
Kabir
user
kabear
6711de895b7360457e26e002
Yang You
6711de895b7360457e26e003
Michael Shieh
admin_assigned
2024-10-18T09:56:45.883Z
64d48e9ef2a3941669810f23
/avatars/495e6785abc44d1facfcf385edd6a222.svg
Michael Shieh
user
Michael2008S
MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
Perceiving and generating diverse modalities are crucial for AI models to
effectively learn from and engage with real-world signals, necessitating
reliable evaluations for their development. We identify two major issues in
current evaluations: (1) inconsistent standards, shaped by different
communities with varying protocols and maturity levels; and (2) significant
query, grading, and generalization biases. To address these, we introduce
MixEval-X, the first any-to-any real-world benchmark designed to optimize and
standardize evaluations across input and output modalities. We propose
multi-modal benchmark mixture and adaptation-rectification pipelines to
reconstruct real-world task distributions, ensuring evaluations generalize
effectively to real-world use cases. Extensive meta-evaluations show our
approach effectively aligns benchmark samples with real-world task
distributions and the model rankings correlate strongly with that of
crowd-sourced real-world evaluations (up to 0.98). We provide comprehensive
leaderboards to rerank existing models and organizations and offer insights to
enhance understanding of multi-modal evaluations and inform future research.
MixEval-X is a benchmark designed to optimize and standardize multi-modal evaluations, addressing issues of inconsistency and bias in real-world applications.
any-to-any real-world benchmark multi-modal benchmark mixture adaptation-rectification pipelines real-world task distributions