Reranking-based Generation for Unbiased Perspective Summarization
Generating unbiased summaries in real-world settings such as political
perspective summarization remains a crucial application of Large Language
Models (LLMs). Yet, existing evaluation frameworks rely on traditional metrics
for measuring key attributes such as coverage and faithfulness without
verifying their applicability, and efforts to develop improved summarizers are
still nascent. We address these gaps by (1) identifying reliable metrics for
measuring perspective summary quality, and (2) investigating the efficacy of
LLM-based methods beyond zero-shot inference. Namely, we build a test set for
benchmarking metric reliability using human annotations and show that
traditional metrics underperform compared to language model-based metrics,
which prove to be strong evaluators. Using these metrics, we show that
reranking-based methods yield strong results, and preference tuning with
synthetically generated and reranking-labeled data further boosts performance.
Our findings aim to contribute to the reliable evaluation and development of
perspective summarization methods.
6858c714c0c8e29df8ea3c9c
Narutatsu Ri
claimed_verified
2025-06-23T08:14:39.250Z
64698ed0dcbb937d56b9dd02
/avatars/407afd70356f9d0669d4e2d39b9a8740.svg
Narutatsu Ri
user
narutatsuri
6858c714c0c8e29df8ea3c9d
Nicholas Deas
6858c714c0c8e29df8ea3c9e
Kathleen McKeown
Reranking-based Generation for Unbiased Perspective Summarization
Generating unbiased summaries in real-world settings such as political
perspective summarization remains a crucial application of Large Language
Models (LLMs). Yet, existing evaluation frameworks rely on traditional metrics
for measuring key attributes such as coverage and faithfulness without
verifying their applicability, and efforts to develop improved summarizers are
still nascent. We address these gaps by (1) identifying reliable metrics for
measuring perspective summary quality, and (2) investigating the efficacy of
LLM-based methods beyond zero-shot inference. Namely, we build a test set for
benchmarking metric reliability using human annotations and show that
traditional metrics underperform compared to language model-based metrics,
which prove to be strong evaluators. Using these metrics, we show that
reranking-based methods yield strong results, and preference tuning with
synthetically generated and reranking-labeled data further boosts performance.
Our findings aim to contribute to the reliable evaluation and development of
perspective summarization methods.
Reranking and preference tuning improve the quality of perspective summaries generated by LLMs, as measured by language model-based metrics that outperform traditional ones.
Large Language Models perspective summarization coverage faithfulness metric reliability reranking-based methods preference tuning synthetically generated data reranking-labeled data