HelloBench: Evaluating Long Text Generation Capabilities of Large
  Language Models
In recent years, Large Language Models (LLMs) have demonstrated remarkable
capabilities in various tasks (e.g., long-context understanding), and many
benchmarks have been proposed. However, we observe that long text generation
capabilities are not well investigated. Therefore, we introduce the
Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive,
in-the-wild, and open-ended benchmark to evaluate LLMs' performance in
generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long
text generation tasks into five subtasks: open-ended QA, summarization, chat,
text completion, and heuristic text generation. Besides, we propose
Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation
method that significantly reduces the time and effort required for human
evaluation while maintaining a high correlation with human evaluation. We have
conducted extensive experiments across around 30 mainstream LLMs and observed
that the current LLMs lack long text generation capabilities. Specifically,
first, regardless of whether the instructions include explicit or implicit
length constraints, we observe that most LLMs cannot generate text that is
longer than 4000 words. Second, we observe that while some LLMs can generate
longer text, many issues exist (e.g., severe repetition and quality
degradation). Third, to demonstrate the effectiveness of HelloEval, we compare
HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge
methods, which show that HelloEval has the highest correlation with human
evaluation. We release our code in https://github.com/Quehry/HelloBench.
66f37983685ee4f1ba09e0f1
Haoran Que
claimed_verified
2024-09-25T07:32:32.953Z
632176664204950905b64e58
https://cdn-avatars.huggingface.co/v1/production/uploads/632176664204950905b64e58/IzkGIaAWy9285Q8uBncZ6.jpeg
quehry
user
quehry
66f37983685ee4f1ba09e0f2
Feiyu Duan
66f37983685ee4f1ba09e0f3
Liqun He
admin_assigned
2024-09-25T18:35:59.010Z
64474bd91f308334030d086a
https://cdn-avatars.huggingface.co/v1/production/uploads/64474bd91f308334030d086a/TkyI2tfOMEYF9Sdja20Of.jpeg
Liqun He
user
liqunhe27
66f37983685ee4f1ba09e0f4
Yutao Mou
66f37983685ee4f1ba09e0f5
Wangchunshu Zhou
admin_assigned
2024-09-25T18:36:11.181Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
66f37983685ee4f1ba09e0f6
Jiaheng Liu
admin_assigned
2024-09-25T18:36:17.540Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
66f37983685ee4f1ba09e0f7
Wenge Rong
66f37983685ee4f1ba09e0f8
Zekun Moore Wang
admin_assigned
2024-09-25T18:36:30.673Z
6149a9e95347647e6bb68882
https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
Zekun Moore Wang
user
ZenMoore
66f37983685ee4f1ba09e0f9
Jian Yang
claimed_verified
2024-10-14T19:01:14.371Z
64ccb9bfead94891d12aef42
/avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
Yang Jian
user
CSJianYang
66f37983685ee4f1ba09e0fa
Ge Zhang
claimed_verified
2024-09-25T07:32:35.486Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66f37983685ee4f1ba09e0fb
Junran Peng
admin_assigned
2024-09-25T18:36:39.841Z
62c11a7360edb2dd7765b80d
/avatars/92db3258d0ba44daee22952f0644cd93.svg
Junran Peng
user
JrPeng
66f37983685ee4f1ba09e0fc
Zhaoxiang Zhang
admin_assigned
2024-09-25T18:36:50.394Z
6350d59989def14ad21e11b3
/avatars/9a7ae6131e0a89f3461f267ee844b218.svg
z
user
smallbert
66f37983685ee4f1ba09e0fd
Songyang Zhang
claimed_verified
2024-09-25T07:32:28.146Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
66f37983685ee4f1ba09e0fe
Kai Chen
HelloBench: Evaluating Long Text Generation Capabilities of Large
  Language Models
In recent years, Large Language Models (LLMs) have demonstrated remarkable
capabilities in various tasks (e.g., long-context understanding), and many
benchmarks have been proposed. However, we observe that long text generation
capabilities are not well investigated. Therefore, we introduce the
Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive,
in-the-wild, and open-ended benchmark to evaluate LLMs' performance in
generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long
text generation tasks into five subtasks: open-ended QA, summarization, chat,
text completion, and heuristic text generation. Besides, we propose
Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation
method that significantly reduces the time and effort required for human
evaluation while maintaining a high correlation with human evaluation. We have
conducted extensive experiments across around 30 mainstream LLMs and observed
that the current LLMs lack long text generation capabilities. Specifically,
first, regardless of whether the instructions include explicit or implicit
length constraints, we observe that most LLMs cannot generate text that is
longer than 4000 words. Second, we observe that while some LLMs can generate
longer text, many issues exist (e.g., severe repetition and quality
degradation). Third, to demonstrate the effectiveness of HelloEval, we compare
HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge
methods, which show that HelloEval has the highest correlation with human
evaluation. We release our code in https://github.com/Quehry/HelloBench.
A comprehensive benchmark, HelloBench, and evaluation method, HelloEval, are introduced to assess the long text generation capabilities of Large Language Models, revealing limitations and providing human-aligned evaluations.
Large Language Models Hierarchical Long Text Generation Benchmark HelloBench Hierarchical Long Text Evaluation HelloEval Bloom's Taxonomy open-ended QA summarization text completion heuristic text generation ROUGE BLEU LLM-as-a-Judge