Dedicated Feedback and Edit Models Empower Inference-Time Scaling for
  Open-Ended General-Domain Tasks
Inference-Time Scaling has been critical to the success of recent models such
as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for
inference-time scaling require tasks to have answers that can be verified,
limiting their application to domains such as math, coding and logical
reasoning. We take inspiration from how humans make first attempts, ask for
detailed feedback from others and make improvements based on such feedback
across a wide spectrum of open-ended endeavors. To this end, we collect data
for and train dedicated Feedback and Edit Models that are capable of performing
inference-time scaling for open-ended general-domain tasks. In our setup, one
model generates an initial response, which are given feedback by a second
model, that are then used by a third model to edit the response. We show that
performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo
can be boosted by scaling the number of initial response drafts, effective
feedback and edited responses. When scaled optimally, our setup based on 70B
models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7
as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and
DeepSeek R1 with 92.3.
67ca637e4cb4283da8ae2979
Zhilin Wang
claimed_verified
2025-03-10T08:03:28.183Z
635865c8aff68f72ac039a4e
/avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
Zhilin Wang
user
zhilinw
67ca637e4cb4283da8ae297a
Jiaqi Zeng
67ca637e4cb4283da8ae297b
Olivier Delalleau
admin_assigned
2025-03-07T09:51:28.127Z
6556379e10428134ff235afd
/avatars/ec569729870d7392e806e59a02f37d0c.svg
Olivier Delalleau
user
odelalleau
67ca637e4cb4283da8ae297c
Daniel Egert
67ca637e4cb4283da8ae297d
Ellie Evans
67ca637e4cb4283da8ae297e
Hoo-Chang Shin
67ca637e4cb4283da8ae297f
Felipe Soares
67ca637e4cb4283da8ae2980
Yi Dong
67ca637e4cb4283da8ae2981
Oleksii Kuchaiev
Dedicated Feedback and Edit Models Empower Inference-Time Scaling for
  Open-Ended General-Domain Tasks
Inference-Time Scaling has been critical to the success of recent models such
as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for
inference-time scaling require tasks to have answers that can be verified,
limiting their application to domains such as math, coding and logical
reasoning. We take inspiration from how humans make first attempts, ask for
detailed feedback from others and make improvements based on such feedback
across a wide spectrum of open-ended endeavors. To this end, we collect data
for and train dedicated Feedback and Edit Models that are capable of performing
inference-time scaling for open-ended general-domain tasks. In our setup, one
model generates an initial response, which are given feedback by a second
model, that are then used by a third model to edit the response. We show that
performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo
can be boosted by scaling the number of initial response drafts, effective
feedback and edited responses. When scaled optimally, our setup based on 70B
models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7
as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and
DeepSeek R1 with 92.3.
Feedback and Edit Models trained for inference-time scaling improve performance on open-ended tasks, surpassing other models on a competitive benchmark.
Inference-Time Scaling Feedback and Edit Models Arena Hard Chatbot Arena Elo Llama 3 parameter-efficient fine-tuning