Complex Logical Instruction Generation
Instruction following has catalyzed the recent era of Large Language Models
(LLMs) and is the foundational skill underpinning more advanced capabilities
such as reasoning and agentic behaviors. As tasks grow more challenging, the
logic structures embedded in natural language instructions becomes increasingly
intricate. However, how well LLMs perform on such logic-rich instructions
remains under-explored. We propose LogicIFGen and LogicIFEval. LogicIFGen is a
scalable, automated framework for generating verifiable instructions from code
functions, which can naturally express rich logic such as conditionals,
nesting, recursion, and function calls. We further curate a collection of
complex code functions and use LogicIFGen to construct LogicIFEval, a benchmark
comprising 426 verifiable logic-rich instructions. Our experiments demonstrate
that current state-of-the-art LLMs still struggle to correctly follow the
instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the
instructions, revealing significant deficiencies in the instruction-following
ability. Code and Benchmark: https://github.com/mianzhang/LogicIF
689ca7d9b083e610d741e88b
Mian Zhang
689ca7d9b083e610d741e88c
Shujian Liu
claimed_verified
2025-08-14T13:38:08.872Z
67d0574cc2ef59a2ab641b80
/avatars/31362733e6b8e745ac3d0586909a7d55.svg
Shujian
user
shujian2025
689ca7d9b083e610d741e88d
Sixun Dong
689ca7d9b083e610d741e88e
Ming Yin
claimed_verified
2025-08-18T06:59:09.575Z
66d4f80bbb76fb26ee6298cf
/avatars/b341445ecc7f9b1c63b4bcba94d83bd0.svg
Ming Yin
user
Kevin355
689ca7d9b083e610d741e88f
Yebowen Hu
claimed_verified
2025-08-29T08:03:41.119Z
61638ee4877577e50b485cd4
https://cdn-avatars.huggingface.co/v1/production/uploads/1633914587203-noauth.jpeg
Yebowen Hu
user
huuuyeah
689ca7d9b083e610d741e890
Xun Wang
689ca7d9b083e610d741e891
Steven Ma
claimed_verified
2025-08-19T10:50:03.955Z
63b8ef18e3c78740d8e85f33
/avatars/debafd7d430f4ce34c4522e4a9bc31ca.svg
Steven Ma
user
mabobbi
689ca7d9b083e610d741e892
Song Wang
claimed_verified
2025-08-26T09:50:50.684Z
624be9911fb0c140d4e8768a
/avatars/fe8a253f2f4fcdf9c60eead7bb90180a.svg
SongW
user
songwang41
689ca7d9b083e610d741e893
Sathish Reddy Indurthi
689ca7d9b083e610d741e894
Haoyun Deng
claimed_verified
2025-08-14T13:38:05.837Z
68524b7b0eaf0b58d631459b
/avatars/e601d2b952f4261ea30294c8ef6de8a6.svg
hd
user
hd-zoom
689ca7d9b083e610d741e895
Zhiyu Zoey Chen
689ca7d9b083e610d741e896
Kaiqiang Song
claimed_verified
2025-08-14T13:38:10.922Z
63a261f13c003e409322c191
/avatars/dc4be6ce1983d0d32620275df124fe45.svg
Kaiqiang Song
user
kqsong
Complex Logical Instruction Generation
Instruction following has catalyzed the recent era of Large Language Models
(LLMs) and is the foundational skill underpinning more advanced capabilities
such as reasoning and agentic behaviors. As tasks grow more challenging, the
logic structures embedded in natural language instructions becomes increasingly
intricate. However, how well LLMs perform on such logic-rich instructions
remains under-explored. We propose LogicIFGen and LogicIFEval. LogicIFGen is a
scalable, automated framework for generating verifiable instructions from code
functions, which can naturally express rich logic such as conditionals,
nesting, recursion, and function calls. We further curate a collection of
complex code functions and use LogicIFGen to construct LogicIFEval, a benchmark
comprising 426 verifiable logic-rich instructions. Our experiments demonstrate
that current state-of-the-art LLMs still struggle to correctly follow the
instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the
instructions, revealing significant deficiencies in the instruction-following
ability. Code and Benchmark: https://github.com/mianzhang/LogicIF
LogicIFGen and LogicIFEval assess the instruction-following capabilities of LLMs on complex, logic-rich instructions, revealing significant performance gaps.
Large Language Models LLMs LogicIFGen LogicIFEval verifiable instructions code functions conditionals nesting recursion function calls benchmark