The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple,
  Self-Contained Directives
Instruction-tuned large language models (IT-LLMs) exhibit strong zero-shot
reasoning, yet their ability to execute simple, self-contained instructions
remains underexplored, despite this being foundational to complex
instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro
benchmarks, by systematically varying the format of option labels (alphabetic,
numeric, Roman) while keeping their meaning identical under four paradigms,
namely: (1) With explicit instructions, label changes cause large performance
shifts (e.g., -30.45\% for Roman vs. numeric), revealing instruction-format
bias. (2) Without instructions, performance drops further (up to -10.84\%) and
label sensitivity intensifies, underscoring the role of explicit guidance. (3)
When option contents are removed, models fail random-choice baselines except
with numeric labels, suggesting weak adherence to atomic directives. (4)
Three-shot exemplars yield no significant gains in robustness or fidelity, and
generation analyses show persistent label errors, especially for non-numeric
formats. Across model sizes, larger LLMs achieve higher accuracy but remain
inconsistent in instruction adherence. These results expose the insufficiencies
of current instruction-tuning paradigms and highlight the need for evaluation
methods and training strategies that explicitly target atomic
instruction-following.
68f6e77a24c44893631117e9
Henry Lim
claimed_verified
2025-10-21T14:04:16.335Z
68f6e73db0f82456fc365d57
/avatars/c3a21b7f28c2eca553cc5da8d0b4fd67.svg
Henry Lim Yu De
user
Henrddy211
68f6e77a24c44893631117ea
Kwan Hui Lim
The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple,
  Self-Contained Directives
Instruction-tuned large language models (IT-LLMs) exhibit strong zero-shot
reasoning, yet their ability to execute simple, self-contained instructions
remains underexplored, despite this being foundational to complex
instruction-following. We evaluate 20 IT-LLMs on modified MMLU and MMLU-Pro
benchmarks, by systematically varying the format of option labels (alphabetic,
numeric, Roman) while keeping their meaning identical under four paradigms,
namely: (1) With explicit instructions, label changes cause large performance
shifts (e.g., -30.45\% for Roman vs. numeric), revealing instruction-format
bias. (2) Without instructions, performance drops further (up to -10.84\%) and
label sensitivity intensifies, underscoring the role of explicit guidance. (3)
When option contents are removed, models fail random-choice baselines except
with numeric labels, suggesting weak adherence to atomic directives. (4)
Three-shot exemplars yield no significant gains in robustness or fidelity, and
generation analyses show persistent label errors, especially for non-numeric
formats. Across model sizes, larger LLMs achieve higher accuracy but remain
inconsistent in instruction adherence. These results expose the insufficiencies
of current instruction-tuning paradigms and highlight the need for evaluation
methods and training strategies that explicitly target atomic
instruction-following.
Evaluation of instruction-tuned large language models on modified MMLU and MMLU-Pro benchmarks reveals significant instruction-format bias and highlights the need for improved atomic instruction-following.
instruction-tuned large language models zero-shot reasoning MMLU MMLU-Pro option labels instruction-format bias explicit instructions three-shot exemplars atomic instruction-following