Spinning the Golden Thread: Benchmarking Long-Form Generation in
  Language Models
The abilities of long-context language models (LMs) are often evaluated using
the "Needle-in-a-Haystack" (NIAH) test, which comprises tasks designed to
assess a model's ability to identify specific information ("needle") within
large text sequences ("haystack"). While these benchmarks measure how well
models understand long-context input sequences, they do not effectively gauge
the quality of long-form text generation--a critical aspect for applications
such as design proposals and creative writing. To address this gap, we have
introduced a new long-form text evaluation benchmark, Spinning the Golden
Thread (SGT), which tests models' ability to identify specific events within
generated long text sequences. In this benchmark, we prompt long-context LMs to
create long-form text that must include particular events or constraints and
evaluate their ability to incorporate these elements. We evaluated ten
long-context LMs across four distinct scenarios, three types of prompt
instructions, and two different generation-length settings (16K and 32K).
Although these models perform well on NIAH benchmarks, none demonstrated
satisfactory performance on the Spinning the Golden Thread, raising concerns
about their ability to generate coherent long-form text that follows
instructions. Additionally, as the length of the generated text increases, all
models exhibit a significant drop in performance.
66df11aeab7f91288b88cc4e
Yuhao Wu
claimed_verified
2024-09-09T19:34:59.092Z
63369da91ba5d5ece24118a4
/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg
wuyuhao
user
mozhu
66df11aeab7f91288b88cc4f
Ming Shan Hee
66df11aeab7f91288b88cc50
Zhiqing Hu
claimed_verified
2024-09-09T19:35:01.146Z
637f228152229c63921119c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
Zhiqiang Hu
user
Zhiqiang007
66df11aeab7f91288b88cc51
Roy Ka-Wei Lee
admin_assigned
2024-09-10T07:12:04.564Z
64b94cc0e3d41dbd6974ae45
/avatars/5edb9d9465addfceccef04c4465a34e6.svg
Roy Ka-Wei lee
user
sroylee
Spinning the Golden Thread: Benchmarking Long-Form Generation in
  Language Models
The abilities of long-context language models (LMs) are often evaluated using
the "Needle-in-a-Haystack" (NIAH) test, which comprises tasks designed to
assess a model's ability to identify specific information ("needle") within
large text sequences ("haystack"). While these benchmarks measure how well
models understand long-context input sequences, they do not effectively gauge
the quality of long-form text generation--a critical aspect for applications
such as design proposals and creative writing. To address this gap, we have
introduced a new long-form text evaluation benchmark, Spinning the Golden
Thread (SGT), which tests models' ability to identify specific events within
generated long text sequences. In this benchmark, we prompt long-context LMs to
create long-form text that must include particular events or constraints and
evaluate their ability to incorporate these elements. We evaluated ten
long-context LMs across four distinct scenarios, three types of prompt
instructions, and two different generation-length settings (16K and 32K).
Although these models perform well on NIAH benchmarks, none demonstrated
satisfactory performance on the Spinning the Golden Thread, raising concerns
about their ability to generate coherent long-form text that follows
instructions. Additionally, as the length of the generated text increases, all
models exhibit a significant drop in performance.
A new benchmark, Spinning the Golden Thread, evaluates long-form text generation by assessing models' ability to incorporate specific events within generated text, revealing performance gaps compared to existing long-context tasks.
long-context language models Needle-in-a-Haystack (NIAH) test long-form text generation Spinning the Golden Thread (SGT) benchmark generation-length settings