Precise Information Control in Long-Form Text Generation
Jacqueline He, Howard Yen, Margaret Li, Stella Li, Zhiyuan Zeng, Weijia Shi, Yulia Tsvetkov, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer
A central challenge in modern language models (LMs) is intrinsic hallucination: the generation of information that is plausible but unsubstantiated relative to input context. To study this problem, we propose Precise Information Control (PIC), a new task formulation that requires models to generate long-form outputs grounded in a provided set of input statements, without adding any unsupported ones. While intrinsic hallucination is commonly evaluated as a binary judgment, we instantiate PIC at the level of short, self-contained statements known as verifiable claims. PIC includes a full setting, which tests a model’s ability to comprehensively include all input claims, and a partial setting, which requires the model to selectively include only relevant claims. We present PIC-Bench, a benchmark of eight long-form generation tasks (e.g., summarization, biography generation) adapted to the PIC setting, where LMs are supplied with well-formed, verifiable input claims. We evaluate a range of open and proprietary models on PIC-Bench; surprisingly, state-of-the-art LMs still intrinsically hallucinate in over 70% of outputs. To alleviate this lack of faithfulness, we introduce a post-training framework, using a weakly supervised preference data construction method, to train an 8B PIC-LM with stronger PIC ability---notably improving from 69.1% to 89.7% F1 in the full setting. When integrated into end-to-end generation pipelines, PIC-LM yields better end-task factual accuracy, improving by 14% in exact match recall on ambiguous QA with retrieval, and by 21% in precision on factual verification tasks, underscoring the broad potential of precisely grounded generation.