AGENTIF: Benchmarking Instruction Following of Large Language Models in
  Agentic Scenarios
Large Language Models (LLMs) have demonstrated advanced capabilities in
real-world agentic applications. Growing research efforts aim to develop
LLM-based agents to address practical demands, introducing a new challenge:
agentic scenarios often involve lengthy instructions with complex constraints,
such as extended system prompts and detailed tool specifications. While
adherence to such instructions is crucial for agentic applications, whether
LLMs can reliably follow them remains underexplored. In this paper, we
introduce AgentIF, the first benchmark for systematically evaluating LLM
instruction following ability in agentic scenarios. AgentIF features three key
characteristics: (1) Realistic, constructed from 50 real-world agentic
applications. (2) Long, averaging 1,723 words with a maximum of 15,630 words.
(3) Complex, averaging 11.9 constraints per instruction, covering diverse
constraint types, such as tool specifications and condition constraints. To
construct AgentIF, we collect 707 human-annotated instructions across 50
agentic tasks from industrial application agents and open-source agentic
systems. For each instruction, we annotate the associated constraints and
corresponding evaluation metrics, including code-based evaluation, LLM-based
evaluation, and hybrid code-LLM evaluation. We use AgentIF to systematically
evaluate existing advanced LLMs. We observe that current models generally
perform poorly, especially in handling complex constraint structures and tool
specifications. We further conduct error analysis and analytical experiments on
instruction length and meta constraints, providing some findings about the
failure modes of existing LLMs. We have released the code and data to
facilitate future research.
683023848d2fd6fc7ce9b99a
Yunjia Qi
683023848d2fd6fc7ce9b99b
Hao Peng
683023848d2fd6fc7ce9b99c
Xiaozhi Wang
683023848d2fd6fc7ce9b99d
Amy Xin
683023848d2fd6fc7ce9b99e
Youfeng Liu
683023848d2fd6fc7ce9b99f
Bin Xu
683023848d2fd6fc7ce9b9a0
Lei Hou
683023848d2fd6fc7ce9b9a1
Juanzi Li
AGENTIF: Benchmarking Instruction Following of Large Language Models in
  Agentic Scenarios
Large Language Models (LLMs) have demonstrated advanced capabilities in
real-world agentic applications. Growing research efforts aim to develop
LLM-based agents to address practical demands, introducing a new challenge:
agentic scenarios often involve lengthy instructions with complex constraints,
such as extended system prompts and detailed tool specifications. While
adherence to such instructions is crucial for agentic applications, whether
LLMs can reliably follow them remains underexplored. In this paper, we
introduce AgentIF, the first benchmark for systematically evaluating LLM
instruction following ability in agentic scenarios. AgentIF features three key
characteristics: (1) Realistic, constructed from 50 real-world agentic
applications. (2) Long, averaging 1,723 words with a maximum of 15,630 words.
(3) Complex, averaging 11.9 constraints per instruction, covering diverse
constraint types, such as tool specifications and condition constraints. To
construct AgentIF, we collect 707 human-annotated instructions across 50
agentic tasks from industrial application agents and open-source agentic
systems. For each instruction, we annotate the associated constraints and
corresponding evaluation metrics, including code-based evaluation, LLM-based
evaluation, and hybrid code-LLM evaluation. We use AgentIF to systematically
evaluate existing advanced LLMs. We observe that current models generally
perform poorly, especially in handling complex constraint structures and tool
specifications. We further conduct error analysis and analytical experiments on
instruction length and meta constraints, providing some findings about the
failure modes of existing LLMs. We have released the code and data to
facilitate future research.
A new benchmark, AgentIF, evaluates Large Language Models' ability to follow complex instructions in realistic agentic scenarios, revealing performance limitations in handling constraints and tool specifications.
AgentIF Large Language Models (LLMs) agentic applications system prompts tool specifications instruction following constraint structures error analysis