Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite
  Learning
Recent studies have identified one aggravating factor of LLM hallucinations
as the knowledge inconsistency between pre-training and fine-tuning, where
unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong
outputs. In this paper, we propose a novel fine-tuning strategy called
Prereq-Tune to address this knowledge inconsistency and reduce hallucinations.
Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge,
so the model learns only the task skills without being impacted by the
knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional
prerequisite learning stage to learn the necessary knowledge for SFT, allowing
subsequent SFT to focus only on task skills. Prereq-Tune can also be combined
with fictitious synthetic data to enhance the grounding of LLM outputs to their
internal knowledge. Experiments show that Prereq-Tune outperforms existing
baselines in improving LLM's factuality across short QA and long-form
generation tasks. It also opens new possibilities for knowledge-controlled
generation in LLMs. Our code is available at
https://github.com/UCSB-NLP-Chang/Prereq_tune.git.
671f0d24941d8e30b6a43180
Yujian Liu
extracted_confirmed
2024-10-28T04:09:03.357Z
615025173e89795099b6fcd9
/avatars/9e2ebea7c5c2d38c95e68af13cc9e382.svg
Yujian Liu
user
yujianll
671f0d24941d8e30b6a43181
Shiyu Chang
extracted_confirmed
2024-10-28T13:57:57.969Z
6696d191d79ce5b27da89c95
/avatars/820c2631c1715a50edb628493656b64f.svg
Shiyu Chang
user
code-terminator
671f0d24941d8e30b6a43182
Tommi Jaakkola
671f0d24941d8e30b6a43183
Yang Zhang
Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite
  Learning
Recent studies have identified one aggravating factor of LLM hallucinations
as the knowledge inconsistency between pre-training and fine-tuning, where
unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong
outputs. In this paper, we propose a novel fine-tuning strategy called
Prereq-Tune to address this knowledge inconsistency and reduce hallucinations.
Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge,
so the model learns only the task skills without being impacted by the
knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional
prerequisite learning stage to learn the necessary knowledge for SFT, allowing
subsequent SFT to focus only on task skills. Prereq-Tune can also be combined
with fictitious synthetic data to enhance the grounding of LLM outputs to their
internal knowledge. Experiments show that Prereq-Tune outperforms existing
baselines in improving LLM's factuality across short QA and long-form
generation tasks. It also opens new possibilities for knowledge-controlled
generation in LLMs. Our code is available at
https://github.com/UCSB-NLP-Chang/Prereq_tune.git.
A novel fine-tuning strategy called Prereq-Tune improves LLM factuality by disentangling skill and knowledge learning and using synthetic data for enhanced grounding.
LLM hallucinations knowledge inconsistency pre-training fine-tuning Prereq-Tune skill learning knowledge learning prerequisite learning stage SFT fictitious synthetic data knowledge-controlled generation