ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
As large language models (LLMs) increasingly interact with external tools,
reward modeling for tool use has become a critical yet underexplored area.
Existing reward models, trained primarily on natural language outputs, struggle
to evaluate tool-based reasoning and execution. To quantify this gap, we
introduce FC-RewardBench, the first benchmark designed to systematically assess
reward models' performance in tool-calling scenarios. Our analysis shows that
current reward models often miss key signals of effective tool use,
highlighting the need for domain-specific modeling. To address this, we propose
a training framework for outcome-based reward models using data synthesized
from permissively licensed, open-weight LLMs. We train models ranging from 1.7B
to 14B parameters and evaluate them across seven out-of-domain benchmarks.
These models consistently outperform general-purpose baselines, achieving up to
25\% average improvement in downstream task performance and enabling
data-efficient fine-tuning through reward-guided filtering.
68c9c4d16e0073c09bd1dd22
Mayank Agarwal
claimed_verified
2025-09-17T13:05:26.269Z
6229237ed94a4a3d5efbacb5
/avatars/7de370e80cc876b33aee79a9959e7af3.svg
Mayank Agarwal
user
mayankagarwal
68c9c4d16e0073c09bd1dd23
Ibrahim Abdelaziz
68c9c4d16e0073c09bd1dd24
Kinjal Basu
claimed_verified
2025-09-17T13:05:23.292Z
64a6c818e3a28de0a4ccc23a
/avatars/0aeca86e12aebbfff7ad1a28691c04ec.svg
Kinjal Basu
user
kinjalbasu
68c9c4d16e0073c09bd1dd25
Merve Unuvar
68c9c4d16e0073c09bd1dd26
Luis A. Lastras
68c9c4d16e0073c09bd1dd27
Yara Rizk
68c9c4d16e0073c09bd1dd28
Pavan Kapanipathi
ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
As large language models (LLMs) increasingly interact with external tools,
reward modeling for tool use has become a critical yet underexplored area.
Existing reward models, trained primarily on natural language outputs, struggle
to evaluate tool-based reasoning and execution. To quantify this gap, we
introduce FC-RewardBench, the first benchmark designed to systematically assess
reward models' performance in tool-calling scenarios. Our analysis shows that
current reward models often miss key signals of effective tool use,
highlighting the need for domain-specific modeling. To address this, we propose
a training framework for outcome-based reward models using data synthesized
from permissively licensed, open-weight LLMs. We train models ranging from 1.7B
to 14B parameters and evaluate them across seven out-of-domain benchmarks.
These models consistently outperform general-purpose baselines, achieving up to
25\% average improvement in downstream task performance and enabling
data-efficient fine-tuning through reward-guided filtering.
A benchmark and training framework for reward models in tool-calling scenarios improve performance and enable efficient fine-tuning through outcome-based evaluation.
large language models LLMs reward modeling tool use FC-RewardBench tool-calling scenarios domain-specific modeling outcome-based reward models open-weight LLMs data-efficient fine-tuning reward-guided filtering