Instruction-Following Evaluation in Function Calling for Large Language
  Models
Function calling is a core capability of large language models, essential for
AI agents. Existing benchmarks such as the Berkeley Function Calling
Leaderboard (BFCL), tau^2-Bench (arXiv:2506.07982), and ACEBench
(arXiv:2501.12851) evaluate argument correctness but do not test adherence to
format instructions embedded in parameter descriptions, such as enclosing
values in double quotes or using ISO date formats.
  We introduce IFEval-FC, a benchmark inspired by IFEval (arXiv:2311.07911)
that assesses precise instruction following in function calling. IFEval-FC
encodes verifiable formats directly within JSON schema descriptions, for
example specifying that a value must not contain punctuation. It includes 750
test cases, each consisting of a function with an embedded format for one of
its input parameters and a corresponding user query. Evaluation is fully
algorithmic, ensuring objectivity, reproducibility, and scalability.
  Our results show that even state-of-the-art proprietary models, including
GPT-5 and Claude 4.1 Opus, frequently fail to follow basic formatting rules,
highlighting a practical limitation for real-world agent systems. The complete
codebase and data are publicly available at
https://github.com/Skripkon/IFEval-FC.
68d7cca60177a6054b013828
Nikolai Skripko
claimed_verified
2025-09-29T06:00:16.219Z
68d7c8e0f2f999edd0cfcbb4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5SyJU2oALWxHj4yKJtDk3.jpeg
Nikolai Skripko
user
NikolaiSkripko
Instruction-Following Evaluation in Function Calling for Large Language
  Models
Function calling is a core capability of large language models, essential for
AI agents. Existing benchmarks such as the Berkeley Function Calling
Leaderboard (BFCL), tau^2-Bench (arXiv:2506.07982), and ACEBench
(arXiv:2501.12851) evaluate argument correctness but do not test adherence to
format instructions embedded in parameter descriptions, such as enclosing
values in double quotes or using ISO date formats.
  We introduce IFEval-FC, a benchmark inspired by IFEval (arXiv:2311.07911)
that assesses precise instruction following in function calling. IFEval-FC
encodes verifiable formats directly within JSON schema descriptions, for
example specifying that a value must not contain punctuation. It includes 750
test cases, each consisting of a function with an embedded format for one of
its input parameters and a corresponding user query. Evaluation is fully
algorithmic, ensuring objectivity, reproducibility, and scalability.
  Our results show that even state-of-the-art proprietary models, including
GPT-5 and Claude 4.1 Opus, frequently fail to follow basic formatting rules,
highlighting a practical limitation for real-world agent systems. The complete
codebase and data are publicly available at
https://github.com/Skripkon/IFEval-FC.
IFEval-FC is a benchmark that evaluates function calling by assessing adherence to format instructions within JSON schema descriptions, revealing that even advanced models often fail to follow basic formatting rules.
function calling large language models AI agents Berkeley Function Calling Leaderboard tau^2-Bench ACEBench IFEval JSON schema descriptions