LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for
  Enhanced Following of Instructions with Multiple Constraints
Instruction following is a key capability for LLMs. However, recent studies
have shown that LLMs often struggle with instructions containing multiple
constraints (e.g. a request to create a social media post "in a funny tone"
with "no hashtag"). Despite this, most evaluations focus solely on synthetic
data. To address this, we introduce RealInstruct, the first benchmark designed
to evaluate LLMs' ability to follow real-world multi-constrained instructions
by leveraging queries real users asked AI assistants. We also investigate
model-based evaluation as a cost-effective alternative to human annotation for
this task. Our findings reveal that even the proprietary GPT-4 model fails to
meet at least one constraint on over 21% of instructions, highlighting the
limitations of state-of-the-art models. To address the performance gap between
open-source and proprietary models, we propose the Decompose, Critique and
Refine (DeCRIM) self-correction pipeline, which enhances LLMs' ability to
follow constraints. DeCRIM works by decomposing the original instruction into a
list of constraints and using a Critic model to decide when and where the LLM's
response needs refinement. Our results show that DeCRIM improves Mistral's
performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
Moreover, we demonstrate that with strong feedback, open-source LLMs with
DeCRIM can outperform GPT-4 on both benchmarks.
6707dec1fd8b5c5c50646e45
Thomas Palmeira Ferraz
admin_assigned
2024-10-10T14:15:37.467Z
60db4e209f0b7223ea4560af
https://cdn-avatars.huggingface.co/v1/production/uploads/1624985089254-noauth.jpeg
Thomas Ferraz
user
thomas-ferraz
6707dec1fd8b5c5c50646e46
Kartik Mehta
claimed_verified
2024-10-11T07:43:59.085Z
610e5127af1badecc0a80e98
/avatars/51a631bec84691de0de6b1ca1bc00f1b.svg
Kartik Mehta
user
kartikmehtaiitd
6707dec1fd8b5c5c50646e47
Yu-Hsiang Lin
6707dec1fd8b5c5c50646e48
Haw-Shiuan Chang
6707dec1fd8b5c5c50646e49
Shereen Oraby
claimed_verified
2025-04-04T07:11:29.441Z
64408bccdc984afcbbbf9ddb
https://cdn-avatars.huggingface.co/v1/production/uploads/64408bccdc984afcbbbf9ddb/0qIkvC8OlyhDHJ0Yjt_tP.jpeg
Shereen Oraby
user
shereenoraby
6707dec1fd8b5c5c50646e4a
Sijia Liu
claimed_verified
2024-10-28T11:25:46.203Z
651d2b485d3519c0b7595af7
/avatars/00ce2ecbc35e22a90f72b9015299aa29.svg
Sijia L
user
sijial430
6707dec1fd8b5c5c50646e4b
Vivek Subramanian
6707dec1fd8b5c5c50646e4c
Tagyoung Chung
6707dec1fd8b5c5c50646e4d
Mohit Bansal
admin_assigned
2024-10-10T14:16:23.705Z
665d9d3a057f7c508f98c625
https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
Mohit Bansal
user
mohitbansal
6707dec1fd8b5c5c50646e4e
Nanyun Peng
claimed_verified
2024-10-11T07:44:01.015Z
64c13de54cbd12e168f09a58
https://cdn-avatars.huggingface.co/v1/production/uploads/64c13de54cbd12e168f09a58/Aw5BaNzf6uuwqQ333OQ5y.png
NANYUN (Violet) PENG
user
violetpeng
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for
  Enhanced Following of Instructions with Multiple Constraints
Instruction following is a key capability for LLMs. However, recent studies
have shown that LLMs often struggle with instructions containing multiple
constraints (e.g. a request to create a social media post "in a funny tone"
with "no hashtag"). Despite this, most evaluations focus solely on synthetic
data. To address this, we introduce RealInstruct, the first benchmark designed
to evaluate LLMs' ability to follow real-world multi-constrained instructions
by leveraging queries real users asked AI assistants. We also investigate
model-based evaluation as a cost-effective alternative to human annotation for
this task. Our findings reveal that even the proprietary GPT-4 model fails to
meet at least one constraint on over 21% of instructions, highlighting the
limitations of state-of-the-art models. To address the performance gap between
open-source and proprietary models, we propose the Decompose, Critique and
Refine (DeCRIM) self-correction pipeline, which enhances LLMs' ability to
follow constraints. DeCRIM works by decomposing the original instruction into a
list of constraints and using a Critic model to decide when and where the LLM's
response needs refinement. Our results show that DeCRIM improves Mistral's
performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
Moreover, we demonstrate that with strong feedback, open-source LLMs with
DeCRIM can outperform GPT-4 on both benchmarks.
RealInstruct evaluates LLMs on real-world instructions with multiple constraints, revealing limitations and introducing DeCRIM, a pipeline that improves their performance.
instruction following RealInstruct multimodal instructions model-based evaluation human annotation Decompose Critique and Refine DeCRIM self-correction pipeline Mistral open-source models GPT-4 benchmark IFEval feedback performance improvement