LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative
  Writing
Evaluating creative writing generated by large language models (LLMs) remains
challenging because open-ended narratives lack ground truths. Without
performant automated evaluation methods, off-the-shelf (OTS) language models
are employed as zero-shot judges, yet their reliability is unclear in this
context. In pursuit of robust evaluation for creative writing, we introduce
LitBench, the first standardized benchmark and paired dataset for creative
writing verification, comprising a held-out test set of 2,480 debiased,
human-labeled story comparisons drawn from Reddit and a 43,827-pair training
corpus of human preference labels. Using LitBench, we (i) benchmark zero-shot
LLM judges, (ii) train Bradley Terry and generative reward models, and (iii)
conduct an online human study to validate reward model rankings on newly
LLM-generated stories. Our benchmark identifies Claude-3.7-Sonnet as the
strongest off-the-shelf judge, reaching 73% agreement with human preferences;
among trained reward models, Bradley-Terry and Generative reward models both
attain an accuracy of 78%, outperforming all off-the-shelf judges. An online
human study further confirms that our trained reward models consistently align
with human preferences in novel LLM-generated stories. We release LitBench and
reward models at
https://huggingface.co/collections/SAA-Lab/litbench-68267b5da3aafe58f9e43461,
providing a vetted resource for reliable, automated evaluation and optimization
of creative writing systems.
6864840ed59a9eda59024a52
Daniel Fein
6864840ed59a9eda59024a53
Sebastian Russo
6864840ed59a9eda59024a54
Violet Xiang
6864840ed59a9eda59024a55
Kabir Jolly
6864840ed59a9eda59024a56
Rafael Rafailov
6864840ed59a9eda59024a57
Nick Haber
LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative
  Writing
Evaluating creative writing generated by large language models (LLMs) remains
challenging because open-ended narratives lack ground truths. Without
performant automated evaluation methods, off-the-shelf (OTS) language models
are employed as zero-shot judges, yet their reliability is unclear in this
context. In pursuit of robust evaluation for creative writing, we introduce
LitBench, the first standardized benchmark and paired dataset for creative
writing verification, comprising a held-out test set of 2,480 debiased,
human-labeled story comparisons drawn from Reddit and a 43,827-pair training
corpus of human preference labels. Using LitBench, we (i) benchmark zero-shot
LLM judges, (ii) train Bradley Terry and generative reward models, and (iii)
conduct an online human study to validate reward model rankings on newly
LLM-generated stories. Our benchmark identifies Claude-3.7-Sonnet as the
strongest off-the-shelf judge, reaching 73% agreement with human preferences;
among trained reward models, Bradley-Terry and Generative reward models both
attain an accuracy of 78%, outperforming all off-the-shelf judges. An online
human study further confirms that our trained reward models consistently align
with human preferences in novel LLM-generated stories. We release LitBench and
reward models at
https://huggingface.co/collections/SAA-Lab/litbench-68267b5da3aafe58f9e43461,
providing a vetted resource for reliable, automated evaluation and optimization
of creative writing systems.
LitBench, a standardized benchmark and dataset, evaluates creative writing by LLMs using zero-shot judges, Bradley Terry, and generative reward models, demonstrating higher accuracy than off-the-shelf judges.
large language models LLMs zero-shot judges LitBench standardized benchmark paired dataset creative writing verification debiased human-labeled story comparisons Reddit human preference labels Bradley Terry generative reward models online human study Claude-3.7-Sonnet