FACTORY: A Challenging Human-Verified Prompt Set for Long-Form
  Factuality
Long-form factuality evaluation assesses the ability of models to generate
accurate, comprehensive responses to short prompts. Existing benchmarks often
lack human verification, leading to potential quality issues. To address this
limitation, we introduce FACTORY, a large-scale, human-verified prompt set.
Developed using a model-in-the-loop approach and refined by humans, FACTORY
includes challenging prompts that are fact-seeking, answerable, and
unambiguous. We conduct human evaluations on 6 state-of-the-art language models
using FACTORY and existing datasets. Our results show that FACTORY is a
challenging benchmark: approximately 40% of the claims made in the responses of
SOTA models are not factual, compared to only 10% for other datasets. Our
analysis identifies the strengths of FACTORY over prior benchmarks, emphasizing
its reliability and the necessity for models to reason across long-tailed
facts.
6894d8f248b0ae5ca2710c5d
Mingda Chen
6894d8f248b0ae5ca2710c5e
Yang Li
6894d8f248b0ae5ca2710c5f
Xilun Chen
6894d8f248b0ae5ca2710c60
Adina Williams
6894d8f248b0ae5ca2710c61
Gargi Ghosh
6894d8f248b0ae5ca2710c62
Scott Yih
FACTORY: A Challenging Human-Verified Prompt Set for Long-Form
  Factuality
Long-form factuality evaluation assesses the ability of models to generate
accurate, comprehensive responses to short prompts. Existing benchmarks often
lack human verification, leading to potential quality issues. To address this
limitation, we introduce FACTORY, a large-scale, human-verified prompt set.
Developed using a model-in-the-loop approach and refined by humans, FACTORY
includes challenging prompts that are fact-seeking, answerable, and
unambiguous. We conduct human evaluations on 6 state-of-the-art language models
using FACTORY and existing datasets. Our results show that FACTORY is a
challenging benchmark: approximately 40% of the claims made in the responses of
SOTA models are not factual, compared to only 10% for other datasets. Our
analysis identifies the strengths of FACTORY over prior benchmarks, emphasizing
its reliability and the necessity for models to reason across long-tailed
facts.
FACTORY, a human-verified prompt set, evaluates the factuality of long-form responses from language models, revealing higher factual accuracy compared to existing datasets.
model-in-the-loop human evaluations state-of-the-art language models long-tailed facts