Teaching Large Language Models to Maintain Contextual Faithfulness via
  Synthetic Tasks and Reinforcement Learning
Teaching large language models (LLMs) to be faithful in the provided context
is crucial for building reliable information-seeking systems. Therefore, we
propose a systematic framework, CANOE, to improve the faithfulness of LLMs in
both short-form and long-form generation tasks without human annotations.
Specifically, we first synthesize short-form question-answering (QA) data with
four diverse tasks to construct high-quality and easily verifiable training
data without human annotation. Also, we propose Dual-GRPO, a rule-based
reinforcement learning method that includes three tailored rule-based rewards
derived from synthesized short-form QA data, while simultaneously optimizing
both short-form and long-form response generation. Notably, Dual-GRPO
eliminates the need to manually label preference data to train reward models
and avoids over-optimizing short-form generation when relying only on the
synthesized short-form QA data. Experimental results show that CANOE greatly
improves the faithfulness of LLMs across 11 different downstream tasks, even
outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.
6833cb27e10e89e250a6d9ae
Shuzheng Si
claimed_verified
2025-05-27T07:55:07.428Z
637c99bbfe115289cfedfb44
https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
ssz
user
ssz1111
6833cb27e10e89e250a6d9af
Haozhe Zhao
claimed_verified
2025-05-26T08:10:14.872Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
6833cb27e10e89e250a6d9b0
Cheng Gao
6833cb27e10e89e250a6d9b1
Yuzhuo Bai
6833cb27e10e89e250a6d9b2
Zhitong Wang
6833cb27e10e89e250a6d9b3
Bofei Gao
6833cb27e10e89e250a6d9b4
Kangyang Luo
6833cb27e10e89e250a6d9b5
Wenhao Li
6833cb27e10e89e250a6d9b6
Yufei Huang
6833cb27e10e89e250a6d9b7
Gang Chen
6833cb27e10e89e250a6d9b8
Fanchao Qi
6833cb27e10e89e250a6d9b9
Minjia Zhang
6833cb27e10e89e250a6d9ba
Baobao Chang
6833cb27e10e89e250a6d9bb
Maosong Sun
Teaching Large Language Models to Maintain Contextual Faithfulness via
  Synthetic Tasks and Reinforcement Learning
Teaching large language models (LLMs) to be faithful in the provided context
is crucial for building reliable information-seeking systems. Therefore, we
propose a systematic framework, CANOE, to improve the faithfulness of LLMs in
both short-form and long-form generation tasks without human annotations.
Specifically, we first synthesize short-form question-answering (QA) data with
four diverse tasks to construct high-quality and easily verifiable training
data without human annotation. Also, we propose Dual-GRPO, a rule-based
reinforcement learning method that includes three tailored rule-based rewards
derived from synthesized short-form QA data, while simultaneously optimizing
both short-form and long-form response generation. Notably, Dual-GRPO
eliminates the need to manually label preference data to train reward models
and avoids over-optimizing short-form generation when relying only on the
synthesized short-form QA data. Experimental results show that CANOE greatly
improves the faithfulness of LLMs across 11 different downstream tasks, even
outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.
CANOE improves LLM faithfulness in generation tasks using synthetic QA data and Dual-GRPO reinforcement learning without human annotations.
teaching large language models faithfulness context CANOE short-form generation long-form generation question-answering Dual-GRPO rule-based reinforcement learning preference data reward models