WildIFEval: Instruction Following in the Wild
Recent LLMs have shown remarkable success in following user instructions, yet
handling instructions with multiple constraints remains a significant
challenge. In this work, we introduce WildIFEval - a large-scale dataset of 12K
real user instructions with diverse, multi-constraint conditions. Unlike prior
datasets, our collection spans a broad lexical and topical spectrum of
constraints, in natural user prompts. We categorize these constraints into
eight high-level classes to capture their distribution and dynamics in
real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments
to benchmark the instruction-following capabilities of leading LLMs. Our
findings reveal that all evaluated models experience performance degradation
with an increasing number of constraints. Thus, we show that all models have a
large room for improvement on such tasks. Moreover, we observe that the
specific type of constraint plays a critical role in model performance. We
release our dataset to promote further research on instruction-following under
complex, realistic conditions.
67d02a147d82f613a31ed396
Gili Lior
admin_assigned
2025-03-14T11:22:05.211Z
62fe0c7d7a4e53610af813a8
/avatars/4188f4275249d10dc6747f63490d0cb6.svg
Gili Lior
user
gililior
67d02a147d82f613a31ed397
Asaf Yehudai
admin_assigned
2025-03-14T11:22:10.848Z
638324f862badff43269e588
/avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
Asaf Yehudai
user
Asaf-Yehudai
67d02a147d82f613a31ed398
Ariel Gera
admin_assigned
2025-03-14T11:22:17.235Z
6415ebd2107962562e9a0712
/avatars/dfd2e6d42c6110bb64b6a73c96eeb3f3.svg
Ariel Gera
user
arielgera
67d02a147d82f613a31ed399
Liat Ein-Dor
WildIFEval: Instruction Following in the Wild
Recent LLMs have shown remarkable success in following user instructions, yet
handling instructions with multiple constraints remains a significant
challenge. In this work, we introduce WildIFEval - a large-scale dataset of 12K
real user instructions with diverse, multi-constraint conditions. Unlike prior
datasets, our collection spans a broad lexical and topical spectrum of
constraints, in natural user prompts. We categorize these constraints into
eight high-level classes to capture their distribution and dynamics in
real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments
to benchmark the instruction-following capabilities of leading LLMs. Our
findings reveal that all evaluated models experience performance degradation
with an increasing number of constraints. Thus, we show that all models have a
large room for improvement on such tasks. Moreover, we observe that the
specific type of constraint plays a critical role in model performance. We
release our dataset to promote further research on instruction-following under
complex, realistic conditions.
WildIFEval, a dataset of 12K real user instructions with multiple constraints, benchmarks LLMs and reveals their performance challenges with increasing constraint complexity.
WildIFEval large-scale dataset real user instructions multi-constraint conditions lexical spectrum topical spectrum natural user prompts high-level classes instruction-following capabilities leading LLMs performance degradation