RePOPE: Impact of Annotation Errors on the POPE Benchmark
Since data annotation is costly, benchmark datasets often incorporate labels
from established image datasets. In this work, we assess the impact of label
errors in MSCOCO on the frequently used object hallucination benchmark POPE. We
re-annotate the benchmark images and identify an imbalance in annotation errors
across different subsets. Evaluating multiple models on the revised labels,
which we denote as RePOPE, we observe notable shifts in model rankings,
highlighting the impact of label quality. Code and data are available at
https://github.com/YanNeu/RePOPE .
6809f6b03e1d48a9bb7f5719
Yannic Neuhaus
claimed_verified
2025-04-24T09:09:36.629Z
650971dbce83a0c12a851000
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ec7H2cZpaZz07W02FT5U9.png
Yannic Neuhaus
user
YanNeu
6809f6b03e1d48a9bb7f571a
Matthias Hein
RePOPE: Impact of Annotation Errors on the POPE Benchmark
Since data annotation is costly, benchmark datasets often incorporate labels
from established image datasets. In this work, we assess the impact of label
errors in MSCOCO on the frequently used object hallucination benchmark POPE. We
re-annotate the benchmark images and identify an imbalance in annotation errors
across different subsets. Evaluating multiple models on the revised labels,
which we denote as RePOPE, we observe notable shifts in model rankings,
highlighting the impact of label quality. Code and data are available at
https://github.com/YanNeu/RePOPE .
Revising labels in the POPE benchmark dataset reveals significant shifts in model performance, emphasizing the importance of label quality.
label errors MSCOCO object hallucination benchmark POPE RePOPE