U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills
  in LLMs
The current evaluation of mathematical skills in LLMs is limited, as existing
benchmarks are either relatively small, primarily focus on elementary and
high-school problems, or lack diversity in topics. Additionally, the inclusion
of visual elements in tasks remains largely under-explored.
  To address these gaps, we introduce U-MATH, a novel benchmark of 1,100
unpublished open-ended university-level problems sourced from teaching
materials. It is balanced across six core subjects, with 20% of multimodal
problems. Given the open-ended nature of U-MATH problems, we employ an LLM to
judge the correctness of generated solutions. To this end, we release
mu-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions.
  The evaluation of general domain, math-specific, and multimodal LLMs
highlights the challenges presented by U-MATH. Our findings reveal that LLMs
achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45%
on visual problems. The solution assessment proves challenging for LLMs, with
the best LLM judge having an F1-score of 80% on mu-MATH.
675156f747111c6a7bcbbdac
Konstantin Chernyshev
claimed_verified
2024-12-05T14:29:22.375Z
6338929107f5708a1aca1390
https://cdn-avatars.huggingface.co/v1/production/uploads/6338929107f5708a1aca1390/Bq_1CDfE8rb7SHTRE8CUU.png
Konstantin Chernyshev 
user
k4black
675156f747111c6a7bcbbdad
Vitaliy Polshkov
claimed_verified
2024-12-05T12:13:59.577Z
650238063e61bc019201e3e2
/avatars/5268b67a116dc24f2b0228c3d87c7cd5.svg
Vitaliy Polshkov
user
cogwheelhead
675156f747111c6a7bcbbdae
Ekaterina Artemova
675156f747111c6a7bcbbdaf
Alex Myasnikov
claimed_verified
2024-12-09T11:07:29.584Z
66da29f728520589aaa1274c
/avatars/8bf855bf3c82bbdfa4241e339feaab75.svg
Alex Myasnikov
user
aseplus
675156f747111c6a7bcbbdb0
Vlad Stepanov
675156f747111c6a7bcbbdb1
Alexei Miasnikov
675156f747111c6a7bcbbdb2
Sergei Tilga
claimed_verified
2024-12-05T12:22:24.344Z
63ef69f4f338cc3b7774422b
/avatars/3975dca73d0941898f637d16e3115872.svg
Sergei Tilga
user
tilgasergey
U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills
  in LLMs
The current evaluation of mathematical skills in LLMs is limited, as existing
benchmarks are either relatively small, primarily focus on elementary and
high-school problems, or lack diversity in topics. Additionally, the inclusion
of visual elements in tasks remains largely under-explored.
  To address these gaps, we introduce U-MATH, a novel benchmark of 1,100
unpublished open-ended university-level problems sourced from teaching
materials. It is balanced across six core subjects, with 20% of multimodal
problems. Given the open-ended nature of U-MATH problems, we employ an LLM to
judge the correctness of generated solutions. To this end, we release
mu-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions.
  The evaluation of general domain, math-specific, and multimodal LLMs
highlights the challenges presented by U-MATH. Our findings reveal that LLMs
achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45%
on visual problems. The solution assessment proves challenging for LLMs, with
the best LLM judge having an F1-score of 80% on mu-MATH.
U-MATH, a new benchmark of university-level math problems, challenges LLMs in both text-based and multimodal tasks, revealing significant limitations in their accuracy and solution assessment capabilities.
benchmark university-level problems multimodal problems LLM general domain math-specific F1-score