LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
The large-scale training of multi-modal models on data scraped from the web
has shown outstanding utility in infusing these models with the required world
knowledge to perform effectively on multiple downstream tasks. However, one
downside of scraping data from the web can be the potential sacrifice of the
benchmarks on which the abilities of these models are often evaluated. To
safeguard against test data contamination and to truly test the abilities of
these foundation models we propose LiveXiv: A scalable evolving live benchmark
based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts
at any given timestamp and proposes to automatically generate visual
question-answer pairs (VQA). This is done without any human-in-the-loop, using
the multi-modal content in the manuscripts, like graphs, charts, and tables.
Moreover, we introduce an efficient evaluation approach that estimates the
performance of all models on the evolving benchmark using evaluations of only a
subset of models. This significantly reduces the overall evaluation cost. We
benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the
first version of our benchmark, showing its challenging nature and exposing the
models true abilities, avoiding contamination. Lastly, in our commitment to
high quality, we have collected and evaluated a manually verified subset. By
comparing its overall results to our automatic annotations, we have found that
the performance variance is indeed minimal (<2.5%). Our dataset is available
online on HuggingFace, and our code will be available here.
670e0604ae37bc424b3bb0c3
Nimrod Shabtay
claimed_verified
2024-10-15T13:06:04.321Z
62bedff7304b82a773bf8c1b
/avatars/f9e79dc196caa95c220127c6212e9944.svg
Nimrod Shabtay
user
NimrodShabtay1986
670e0604ae37bc424b3bb0c4
Felipe Maia Polo
670e0604ae37bc424b3bb0c5
Sivan Doveh
670e0604ae37bc424b3bb0c6
Wei Lin
claimed_verified
2024-10-15T08:07:46.917Z
64b99a3fd4463c3d2183739d
/avatars/c65f9bc024e0e1c0073c86a6421c77c5.svg
Wei Lin
user
wlin21at
670e0604ae37bc424b3bb0c7
M. Jehanzeb Mirza
670e0604ae37bc424b3bb0c8
Leshem Chosen
670e0604ae37bc424b3bb0c9
Mikhail Yurochkin
670e0604ae37bc424b3bb0ca
Yuekai Sun
claimed_verified
2025-05-07T08:10:21.247Z
64ecbb54bb07e4c5a91dc75e
https://cdn-avatars.huggingface.co/v1/production/uploads/64ecbb54bb07e4c5a91dc75e/NBpYlvP1seLxj9co05rBr.png
Yuekai Sun
user
syuekai
670e0604ae37bc424b3bb0cb
Assaf Arbelle
670e0604ae37bc424b3bb0cc
Leonid Karlinsky
670e0604ae37bc424b3bb0cd
Raja Giryes
LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
The large-scale training of multi-modal models on data scraped from the web
has shown outstanding utility in infusing these models with the required world
knowledge to perform effectively on multiple downstream tasks. However, one
downside of scraping data from the web can be the potential sacrifice of the
benchmarks on which the abilities of these models are often evaluated. To
safeguard against test data contamination and to truly test the abilities of
these foundation models we propose LiveXiv: A scalable evolving live benchmark
based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts
at any given timestamp and proposes to automatically generate visual
question-answer pairs (VQA). This is done without any human-in-the-loop, using
the multi-modal content in the manuscripts, like graphs, charts, and tables.
Moreover, we introduce an efficient evaluation approach that estimates the
performance of all models on the evolving benchmark using evaluations of only a
subset of models. This significantly reduces the overall evaluation cost. We
benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the
first version of our benchmark, showing its challenging nature and exposing the
models true abilities, avoiding contamination. Lastly, in our commitment to
high quality, we have collected and evaluated a manually verified subset. By
comparing its overall results to our automatic annotations, we have found that
the performance variance is indeed minimal (<2.5%). Our dataset is available
online on HuggingFace, and our code will be available here.
LiveXiv is a scalable, evolving benchmark using scientific ArXiv papers to generate visual question-answer pairs, providing a contamination-free evaluation for large multi-modal models.
multi-modal models web scraping test data contamination benchmark scientific ArXiv papers visual question-answer pairs graphs charts tables evaluation approach Large Multi-modal Models HuggingFace