IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with
  Verifiable Rewards
Reinforcement Learning with Verifiable Rewards (RLVR) improves instruction
following capabilities of large language models (LLMs), but suffers from
training inefficiency due to inadequate difficulty assessment. Moreover, RLVR
is prone to over-optimization, where LLMs exploit verification shortcuts
without aligning to the actual intent of user instructions. We introduce
Instruction Following Decorator (IFDecorator}, a framework that wraps RLVR
training into a robust and sample-efficient pipeline. It consists of three
components: (1) a cooperative-adversarial data flywheel that co-evolves
instructions and hybrid verifications, generating progressively more
challenging instruction-verification pairs; (2) IntentCheck, a bypass module
enforcing intent alignment; and (3) trip wires, a diagnostic mechanism that
detects reward hacking via trap instructions, which trigger and capture
shortcut exploitation behaviors. Our Qwen2.5-32B-Instruct-IFDecorator achieves
87.43% accuracy on IFEval, outperforming larger proprietary models such as
GPT-4o. Additionally, we demonstrate substantial improvements on FollowBench
while preserving general capabilities. Our trip wires show significant
reductions in reward hacking rates. We will release models, code, and data for
future research.
689458cc741a16f544fbd02e
Xu Guo
claimed_verified
2025-08-22T07:24:57.312Z
6687bdeb3495d4c1c4d0b8a2
/avatars/40b229b809347f83d7c0f694ee2b693a.svg
guox18
user
guox18
689458cc741a16f544fbd02f
Tianyi Liang
claimed_verified
2025-08-08T16:21:38.729Z
62c14609ac1b639c2d87192c
https://cdn-avatars.huggingface.co/v1/production/uploads/1656833489364-noauth.png
liangtianyi
user
tianyilt
689458cc741a16f544fbd030
Tong Jian
689458cc741a16f544fbd031
Xiaogui Yang
689458cc741a16f544fbd032
Ling-I Wu
689458cc741a16f544fbd033
Chenhui Li
689458cc741a16f544fbd034
Zhihui Lu
689458cc741a16f544fbd035
Qipeng Guo
689458cc741a16f544fbd036
Kai Chen
IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with
  Verifiable Rewards
Reinforcement Learning with Verifiable Rewards (RLVR) improves instruction
following capabilities of large language models (LLMs), but suffers from
training inefficiency due to inadequate difficulty assessment. Moreover, RLVR
is prone to over-optimization, where LLMs exploit verification shortcuts
without aligning to the actual intent of user instructions. We introduce
Instruction Following Decorator (IFDecorator}, a framework that wraps RLVR
training into a robust and sample-efficient pipeline. It consists of three
components: (1) a cooperative-adversarial data flywheel that co-evolves
instructions and hybrid verifications, generating progressively more
challenging instruction-verification pairs; (2) IntentCheck, a bypass module
enforcing intent alignment; and (3) trip wires, a diagnostic mechanism that
detects reward hacking via trap instructions, which trigger and capture
shortcut exploitation behaviors. Our Qwen2.5-32B-Instruct-IFDecorator achieves
87.43% accuracy on IFEval, outperforming larger proprietary models such as
GPT-4o. Additionally, we demonstrate substantial improvements on FollowBench
while preserving general capabilities. Our trip wires show significant
reductions in reward hacking rates. We will release models, code, and data for
future research.
Instruction Following Decorator enhances RLVR by improving sample efficiency, intent alignment, and reducing reward hacking in large language models.
Reinforcement Learning with Verifiable Rewards RLVR large language models LLMs cooperative-adversarial data flywheel hybrid verifications IntentCheck trip wires reward hacking IFEval FollowBench