Learning from Failures in Multi-Attempt Reinforcement Learning
Recent advancements in reinforcement learning (RL) for large language models
(LLMs), exemplified by DeepSeek R1, have shown that even a simple
question-answering task can substantially improve an LLM's reasoning
capabilities. In this work, we extend this approach by modifying the task into
a multi-attempt setting. Instead of generating a single response per question,
the model is given multiple attempts, with feedback provided after incorrect
responses. The multi-attempt task encourages the model to refine its previous
attempts and improve search efficiency. Experimental results show that even a
small LLM trained on a multi-attempt task achieves significantly higher
accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt
to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM
trained on a standard single-turn task exhibits only a marginal improvement,
increasing from 42.3% to 43.2% when given more attempts during evaluation. The
results indicate that, compared to the standard single-turn task, an LLM
trained on a multi-attempt task achieves slightly better performance on math
benchmarks while also learning to refine its responses more effectively based
on user feedback. Full code is available at
https://github.com/DualityRL/multi-attempt
67ce5c7065b141ae6b0d3957
Stephen Chung
claimed_verified
2025-03-10T17:39:13.472Z
67cef8b7d9f3ce4930069e10
/avatars/0b660e5ff13cf6cc27804bc722a70efc.svg
Sephen Chung
user
stephenchungmh
67ce5c7065b141ae6b0d3958
Wenyu Du
admin_assigned
2025-03-10T10:27:00.543Z
624c3d2ca19f20b197761ba9
/avatars/7a64b81c29f4f6700fa18effc5616865.svg
Wenyu Du
user
wydu
67ce5c7065b141ae6b0d3959
Jie Fu
claimed_verified
2025-03-10T12:46:28.886Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
Learning from Failures in Multi-Attempt Reinforcement Learning
Recent advancements in reinforcement learning (RL) for large language models
(LLMs), exemplified by DeepSeek R1, have shown that even a simple
question-answering task can substantially improve an LLM's reasoning
capabilities. In this work, we extend this approach by modifying the task into
a multi-attempt setting. Instead of generating a single response per question,
the model is given multiple attempts, with feedback provided after incorrect
responses. The multi-attempt task encourages the model to refine its previous
attempts and improve search efficiency. Experimental results show that even a
small LLM trained on a multi-attempt task achieves significantly higher
accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt
to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM
trained on a standard single-turn task exhibits only a marginal improvement,
increasing from 42.3% to 43.2% when given more attempts during evaluation. The
results indicate that, compared to the standard single-turn task, an LLM
trained on a multi-attempt task achieves slightly better performance on math
benchmarks while also learning to refine its responses more effectively based
on user feedback. Full code is available at
https://github.com/DualityRL/multi-attempt
Training large language models on a multi-attempt task with feedback improves their accuracy and response refinement compared to standard single-turn tasks.
reinforcement learning large language models multi-attempt setting feedback search efficiency math benchmark