Beyond In-Context Learning: Aligning Long-form Generation of Large
  Language Models via Task-Inherent Attribute Guidelines
In-context learning (ICL) is an important yet not fully understood ability of
pre-trained large language models (LLMs). It can greatly enhance task
performance using a few examples, termed demonstrations, without fine-tuning.
Although effective in question answering, ICL often underperforms in long-form
generation tasks such as summarization. Under appropriately realistic
assumptions, we empirically and theoretically show that ICL demonstrations
alone are insufficient to teach LLMs the task language and format distributions
for generation. We argue for explicit exposure to the task distributions and
hypothesize that defining them by prompting enhances model performance. To this
end, we present LongGuide, which efficiently generates two parallel streams of
guidelines capturing task language and format properties: (i) Metric Guidelines
(MGs) that instruct models to optimize self-evaluated metrics; and (ii) Output
Constraint Guidelines (OCGs) that constrain generation at both token and
sentence levels. LongGuide automatically selects the best combination of
guidelines, improving both strong open- and closed-source LLMs by over 5% in
both zero- and few-shot settings. We show that LongGuide is generalizable,
learnable by weak models to enhance strong ones, and integrates synergistically
with automatic prompt optimizers.
68406f484024b3c937eebb9b
Do Xuan Long
claimed_verified
2025-06-05T07:27:41.485Z
63a9a0d13453852ef53c0b37
/avatars/411c4ffc2a3e89047a23c6e7442f6ed5.svg
Do Xuan Long
user
dxlong2000
68406f484024b3c937eebb9c
Duong Ngoc Yen
68406f484024b3c937eebb9d
Do Xuan Trong
68406f484024b3c937eebb9e
Luu Anh Tuan
68406f484024b3c937eebb9f
Kenji Kawaguchi
68406f484024b3c937eebba0
Shafiq Joty
68406f484024b3c937eebba1
Min-Yen Kan
admin_assigned
2025-06-05T07:34:41.342Z
628f137053028c04a3b750e4
/avatars/85d174cc041ab91e724bb9d9d4e46c88.svg
Min-Yen Kan
user
knmnyn
68406f484024b3c937eebba2
Nancy F. Chen
admin_assigned
2025-06-05T07:34:35.789Z
6774ab7a1f7f8da50570ec29
https://cdn-avatars.huggingface.co/v1/production/uploads/6774ab7a1f7f8da50570ec29/5w71bqq_w3QxWbb1FLyto.jpeg
Nancy F. Chen
user
nancy-f-chen
Beyond In-Context Learning: Aligning Long-form Generation of Large
  Language Models via Task-Inherent Attribute Guidelines
In-context learning (ICL) is an important yet not fully understood ability of
pre-trained large language models (LLMs). It can greatly enhance task
performance using a few examples, termed demonstrations, without fine-tuning.
Although effective in question answering, ICL often underperforms in long-form
generation tasks such as summarization. Under appropriately realistic
assumptions, we empirically and theoretically show that ICL demonstrations
alone are insufficient to teach LLMs the task language and format distributions
for generation. We argue for explicit exposure to the task distributions and
hypothesize that defining them by prompting enhances model performance. To this
end, we present LongGuide, which efficiently generates two parallel streams of
guidelines capturing task language and format properties: (i) Metric Guidelines
(MGs) that instruct models to optimize self-evaluated metrics; and (ii) Output
Constraint Guidelines (OCGs) that constrain generation at both token and
sentence levels. LongGuide automatically selects the best combination of
guidelines, improving both strong open- and closed-source LLMs by over 5% in
both zero- and few-shot settings. We show that LongGuide is generalizable,
learnable by weak models to enhance strong ones, and integrates synergistically
with automatic prompt optimizers.
LongGuide enhances pre-trained LLMs in long-form generation tasks by providing metric and output constraint guidelines, improving model performance in zero- and few-shot settings.
in-context learning pre-trained large language models task performance demonstrations long-form generation summarization ICL task language format distributions Metric Guidelines Output Constraint Guidelines zero-shot few-shot