TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue
  Summarization
Single document news summarization has seen substantial progress on
faithfulness in recent years, driven by research on the evaluation of factual
consistency, or hallucinations. We ask whether these advances carry over to
other text summarization domains. We propose a new evaluation benchmark on
topic-focused dialogue summarization, generated by LLMs of varying sizes. We
provide binary sentence-level human annotations of the factual consistency of
these summaries along with detailed explanations of factually inconsistent
sentences. Our analysis shows that existing LLMs hallucinate significant
amounts of factual errors in the dialogue domain, regardless of the model's
size. On the other hand, when LLMs, including GPT-4, serve as binary factual
evaluators, they perform poorly and can be outperformed by prevailing
state-of-the-art specialized factuality evaluation metrics. Finally, we
conducted an analysis of hallucination types with a curated error taxonomy. We
find that there are diverse errors and error distributions in model-generated
summaries and that non-LLM based metrics can capture all error types better
than LLM-based evaluators.
65d570a79936544a08a0837a
Liyan Tang
claimed_verified
2024-02-23T16:36:13.785Z
62c70672e7d825deaae41e5e
https://cdn-avatars.huggingface.co/v1/production/uploads/62c70672e7d825deaae41e5e/ICCpeBwmQ1NsgWcjG-MEZ.png
Liyan Tang
user
lytang
65d570a79936544a08a0837b
Igor Shalyminov
admin_assigned
2024-02-21T10:43:21.082Z
61fa7a12620023c7e77a096f
/avatars/dc7e5aded5eb47c937880675a531ab47.svg
Shalyminov
user
ishalyminov
65d570a79936544a08a0837c
Amy Wing-mei Wong
65d570a79936544a08a0837d
Jon Burnsky
65d570a79936544a08a0837e
Jake W. Vincent
admin_assigned
2024-02-21T10:40:42.864Z
64a3695aae76a4a3673e91a5
/avatars/7180f90f37e412ba573990c136110944.svg
Jake Vincent
user
jakewvincent
65d570a79936544a08a0837f
Yu'an Yang
65d570a79936544a08a08380
Siffi Singh
admin_assigned
2024-02-21T10:41:14.904Z
63988cf124e61e0d386f22a3
/avatars/a39d485f8a0ead0e74ad6d5a612e7908.svg
Siffi Singh
user
siffi26
65d570a79936544a08a08381
Song Feng
65d570a79936544a08a08382
Hwanjun Song
admin_assigned
2024-02-21T10:45:58.484Z
63c9da8d5fdc575773c84816
/avatars/9357452f7df22c98a3ac6dbb90aa1b14.svg
Song
user
Hwanjun
65d570a79936544a08a08383
Hang Su
65d570a79936544a08a08384
Lijia Sun
65d570a79936544a08a08385
Yi Zhang
65d570a79936544a08a08386
Saab Mansour
65d570a79936544a08a08387
Kathleen McKeown
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue
  Summarization
Single document news summarization has seen substantial progress on
faithfulness in recent years, driven by research on the evaluation of factual
consistency, or hallucinations. We ask whether these advances carry over to
other text summarization domains. We propose a new evaluation benchmark on
topic-focused dialogue summarization, generated by LLMs of varying sizes. We
provide binary sentence-level human annotations of the factual consistency of
these summaries along with detailed explanations of factually inconsistent
sentences. Our analysis shows that existing LLMs hallucinate significant
amounts of factual errors in the dialogue domain, regardless of the model's
size. On the other hand, when LLMs, including GPT-4, serve as binary factual
evaluators, they perform poorly and can be outperformed by prevailing
state-of-the-art specialized factuality evaluation metrics. Finally, we
conducted an analysis of hallucination types with a curated error taxonomy. We
find that there are diverse errors and error distributions in model-generated
summaries and that non-LLM based metrics can capture all error types better
than LLM-based evaluators.
Existing large language models generate factually inconsistent topic-focused dialogue summaries and perform poorly as binary factual evaluators, whereas specialized non-LLM metrics capture diverse hallucination types more effectively.
LLMs topic-focused dialogue summarization factual consistency hallucinations binary sentence-level human annotations GPT-4 binary factual evaluators specialized factuality evaluation metrics error taxonomy