LEMMA: Learning from Errors for MatheMatical Advancement in LLMs
Large language models (LLMs) have demonstrated remarkable reasoning
capability in solving mathematical problems. However, existing approaches
primarily focus on improving the quality of correct training data, e.g.,
distilling high-quality correct solutions from advanced models, neglecting the
value contained in error data, potentially hindering the model's reflective
ability. Though some studies attempt to leverage error data, they often involve
complex mechanisms, such as Monte Carlo Tree Search (MCTS) to explore error
nodes. In this work, we propose to enhance LLMs' reasoning ability by Learning
from Errors for Mathematical Advancement (LEMMA). LEMMA constructs data
consisting of an incorrect solution with an erroneous step and a reflection
connection to a correct solution for fine-tuning. Specifically, we
systematically analyze the model-generated error types and introduce an
error-type grounded mistake augmentation method to collect diverse and
representative errors. Correct solutions are either from fixing the errors or
generating a fresh start. Through a model-aware smooth reflection connection,
the erroneous solution is transferred to the correct one. By fine-tuning on the
constructed dataset, the model is able to self-correct errors autonomously
within the generation process without relying on external critique models.
Experimental results demonstrate that LEMMA achieves significant performance
improvements over other strong baselines.
67e21f63fb4213c53714be08
Zhuoshi Pan
claimed_verified
2025-07-15T19:12:53.189Z
6565e44b9bf6665f10016213
/avatars/0d7cf0e1b42cd116960cd030478446c5.svg
Zhuoshi Pan
user
panzs19
67e21f63fb4213c53714be09
Yu Li
claimed_verified
2025-04-29T08:00:04.396Z
671b852aa4fa4f8f5fb5404c
https://cdn-avatars.huggingface.co/v1/production/uploads/671b852aa4fa4f8f5fb5404c/TDLsgP8WgKW-qaA8Ys-iJ.jpeg
YU LI
user
yu0226
67e21f63fb4213c53714be0a
Honglin Lin
admin_assigned
2025-03-25T09:17:30.635Z
640d99628512ec51d7ef71c7
https://cdn-avatars.huggingface.co/v1/production/uploads/640d99628512ec51d7ef71c7/fcBkqnxfxuuuZTqfN_BGy.jpeg
Honglin Lin
user
LHL3341
67e21f63fb4213c53714be0b
Qizhi Pei
admin_assigned
2025-03-25T09:17:42.836Z
6397f6081323f19c578f142e
https://cdn-avatars.huggingface.co/v1/production/uploads/6397f6081323f19c578f142e/it7FYYKjlLX8wSsMLm8EO.jpeg
QizhiPei
user
QizhiPei
67e21f63fb4213c53714be0c
Zinan Tang
admin_assigned
2025-03-25T09:17:50.791Z
66580d3d80ee5b1e11a94e57
https://cdn-avatars.huggingface.co/v1/production/uploads/66580d3d80ee5b1e11a94e57/aHaPqrV5vNefFktYRsiGf.jpeg
Zinan Tang
user
Word2Li
67e21f63fb4213c53714be0d
Wei Wu
67e21f63fb4213c53714be0e
Chenlin Ming
admin_assigned
2025-03-25T09:18:12.485Z
677e133ee86d0754dc7ce296
/avatars/c16511c1876b50c2d049925c5f320d15.svg
mingchenlin
user
mingchenlin2025
67e21f63fb4213c53714be0f
H. Vicky Zhao
67e21f63fb4213c53714be10
Conghui He
admin_assigned
2025-03-25T09:18:25.368Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
67e21f63fb4213c53714be11
Lijun Wu
LEMMA: Learning from Errors for MatheMatical Advancement in LLMs
Large language models (LLMs) have demonstrated remarkable reasoning
capability in solving mathematical problems. However, existing approaches
primarily focus on improving the quality of correct training data, e.g.,
distilling high-quality correct solutions from advanced models, neglecting the
value contained in error data, potentially hindering the model's reflective
ability. Though some studies attempt to leverage error data, they often involve
complex mechanisms, such as Monte Carlo Tree Search (MCTS) to explore error
nodes. In this work, we propose to enhance LLMs' reasoning ability by Learning
from Errors for Mathematical Advancement (LEMMA). LEMMA constructs data
consisting of an incorrect solution with an erroneous step and a reflection
connection to a correct solution for fine-tuning. Specifically, we
systematically analyze the model-generated error types and introduce an
error-type grounded mistake augmentation method to collect diverse and
representative errors. Correct solutions are either from fixing the errors or
generating a fresh start. Through a model-aware smooth reflection connection,
the erroneous solution is transferred to the correct one. By fine-tuning on the
constructed dataset, the model is able to self-correct errors autonomously
within the generation process without relying on external critique models.
Experimental results demonstrate that LEMMA achieves significant performance
improvements over other strong baselines.
LEMMA enhances LLMs' reasoning by fine-tuning on error data with reflection connections to correct solutions, showcasing performance improvements over existing methods.
large language models LLMs mathematical problems training data error data reflection ability Monte Carlo Tree Search MCTS Learning from Errors for Mathematical Advancement LEMMA mistake augmentation correct solutions reflection connection self-correction generation process baseline methods