LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
Current long context large language models (LLMs) can process inputs up to
100,000 tokens, yet struggle to generate outputs exceeding even a modest length
of 2,000 words. Through controlled experiments, we find that the model's
effective generation length is inherently bounded by the sample it has seen
during supervised fine-tuning (SFT). In other words, their output limitation is
due to the scarcity of long-output examples in existing SFT datasets. To
address this, we introduce AgentWrite, an agent-based pipeline that decomposes
ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to
generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we
construct LongWriter-6k, a dataset containing 6,000 SFT data with output
lengths ranging from 2k to 32k words. By incorporating this dataset into model
training, we successfully scale the output length of existing models to over
10,000 words while maintaining output quality. We also develop LongBench-Write,
a comprehensive benchmark for evaluating ultra-long generation capabilities.
Our 9B parameter model, further improved through DPO, achieves state-of-the-art
performance on this benchmark, surpassing even much larger proprietary models.
In general, our work demonstrates that existing long context LLM already
possesses the potential for a larger output window--all you need is data with
extended output during model alignment to unlock this capability. Our code &
models are at: https://github.com/THUDM/LongWriter.
66bc1166d56c232066b9e717
Yushi Bai
claimed_verified
2024-08-14T08:11:32.209Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
66bc1166d56c232066b9e718
Jiajie Zhang
claimed_verified
2024-09-04T07:25:35.184Z
66cdd285c51a915bd5f2d017
/avatars/14e5794307e4672b1b51d26b31227e0f.svg
Jiajie Zhang
user
NeoZ123
66bc1166d56c232066b9e719
Xin Lv
claimed_verified
2024-08-14T08:11:29.883Z
648c64829c935db2b527a764
https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
Xin Lv
user
davidlvxin
66bc1166d56c232066b9e71a
Linzhi Zheng
admin_assigned
2024-08-14T08:59:55.458Z
66a99e585d7a1151f2cc41fe
/avatars/c79feaa920e66864090358716eb0133c.svg
Linzhi Zheng
user
linzhizheng
66bc1166d56c232066b9e71b
Siqi Zhu
claimed_verified
2024-08-29T15:22:14.840Z
65621fd68631d43d2baf33b2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1JemNCnPkS1mE3SNygsE2.png
siqi zhu
user
zsqzz
66bc1166d56c232066b9e71c
Lei Hou
66bc1166d56c232066b9e71d
Yuxiao Dong
admin_assigned
2024-08-14T10:34:59.607Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
66bc1166d56c232066b9e71e
Jie Tang
admin_assigned
2024-08-14T10:34:53.801Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
66bc1166d56c232066b9e71f
Juanzi Li
admin_assigned
2024-08-14T10:34:46.893Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
Current long context large language models (LLMs) can process inputs up to
100,000 tokens, yet struggle to generate outputs exceeding even a modest length
of 2,000 words. Through controlled experiments, we find that the model's
effective generation length is inherently bounded by the sample it has seen
during supervised fine-tuning (SFT). In other words, their output limitation is
due to the scarcity of long-output examples in existing SFT datasets. To
address this, we introduce AgentWrite, an agent-based pipeline that decomposes
ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to
generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we
construct LongWriter-6k, a dataset containing 6,000 SFT data with output
lengths ranging from 2k to 32k words. By incorporating this dataset into model
training, we successfully scale the output length of existing models to over
10,000 words while maintaining output quality. We also develop LongBench-Write,
a comprehensive benchmark for evaluating ultra-long generation capabilities.
Our 9B parameter model, further improved through DPO, achieves state-of-the-art
performance on this benchmark, surpassing even much larger proprietary models.
In general, our work demonstrates that existing long context LLM already
possesses the potential for a larger output window--all you need is data with
extended output during model alignment to unlock this capability. Our code &
models are at: https://github.com/THUDM/LongWriter.
AgentWrite enables long context LLMs to generate outputs exceeding 20,000 words by decomposing tasks into subtasks, using a new dataset to train models with extended output capabilities.
LLMs long context tokens AgentWrite subtasks LongWriter-6k long-output examples SFT datasets output quality LongBench-Write DPO