Linguistic Generalizability of Test-Time Scaling in Mathematical
  Reasoning
Scaling pre-training compute has proven effective for achieving
mulitlinguality, but does the same hold for test-time scaling? In this work, we
introduce MCLM, a multilingual math benchmark featuring competition-level
problems in 55 languages. We test three test-time scaling methods-Outcome
Reward Modeling (ORM), Process Reward Modeling (ORM), and Budget Forcing
(BF)-on both Qwen2.5-1.5B Math and MR1-1.5B, a multilingual LLM we trained for
extended reasoning. Our experiments show that using Qwen2.5-1.5B Math with ORM
achieves a score of 35.8 on MCLM, while BF on MR1-1.5B attains 35.2. Although
"thinking LLMs" have recently garnered significant attention, we find that
their performance is comparable to traditional scaling methods like best-of-N
once constrained to similar levels of inference FLOPs. Moreover, while BF
yields a 20-point improvement on English AIME, it provides only a 1.94-point
average gain across other languages-a pattern consistent across the other
test-time scaling methods we studied-higlighting that test-time scaling may not
generalize as effectively to multilingual tasks. To foster further research, we
release MCLM, MR1-1.5B, and evaluation results.
67bd48d4becb766415a5d19d
Guijin Son
admin_assigned
2025-02-25T16:21:47.182Z
60d3e619b8448e1785bbda2a
https://cdn-avatars.huggingface.co/v1/production/uploads/60d3e619b8448e1785bbda2a/q2re5u1HNwsCCyIMtid_I.jpeg
GUIJIN SON
user
amphora
67bd48d4becb766415a5d19e
Jiwoo Hong
admin_assigned
2025-02-25T16:21:56.398Z
678b61b6c7491f8b7065f68d
/avatars/2168e7a0c58076126fcb41b01d01e622.svg
Jiwoo Hong
user
hongmush
67bd48d4becb766415a5d19f
Hyunwoo Ko
claimed_verified
2025-02-25T09:39:12.933Z
63e087b6a98d931aa90c1b9c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e087b6a98d931aa90c1b9c/96c6IT3f1pWGLbRdRDB2U.png
Hyunwoo Ko
user
Cartinoe5930
67bd48d4becb766415a5d1a0
James Thorne
Linguistic Generalizability of Test-Time Scaling in Mathematical
  Reasoning
Scaling pre-training compute has proven effective for achieving
mulitlinguality, but does the same hold for test-time scaling? In this work, we
introduce MCLM, a multilingual math benchmark featuring competition-level
problems in 55 languages. We test three test-time scaling methods-Outcome
Reward Modeling (ORM), Process Reward Modeling (ORM), and Budget Forcing
(BF)-on both Qwen2.5-1.5B Math and MR1-1.5B, a multilingual LLM we trained for
extended reasoning. Our experiments show that using Qwen2.5-1.5B Math with ORM
achieves a score of 35.8 on MCLM, while BF on MR1-1.5B attains 35.2. Although
"thinking LLMs" have recently garnered significant attention, we find that
their performance is comparable to traditional scaling methods like best-of-N
once constrained to similar levels of inference FLOPs. Moreover, while BF
yields a 20-point improvement on English AIME, it provides only a 1.94-point
average gain across other languages-a pattern consistent across the other
test-time scaling methods we studied-higlighting that test-time scaling may not
generalize as effectively to multilingual tasks. To foster further research, we
release MCLM, MR1-1.5B, and evaluation results.
Multilingual math benchmark MCLM tests various test-time scaling methods on multilingual LLMs, showing that performance gains from these methods are limited outside of English.
MCLM Outcome Reward Modeling Process Reward Modeling Budget Forcing Qwen2.5-1.5B Math MR1-1.5B multilingual LLM inference FLOPs AIME