Investigating and Mitigating Catastrophic Forgetting in Medical Knowledge Injection through Internal Knowledge Augmentation Learning
Yuxuan Zhou, Xien Liu, Xiao Zhang, Chen Ning, Shijin Wang, Guoping Hu, Ji Wu
Large Language Models (LLMs) must possess sufficient and comprehensive medical knowledge to effectively address real-world clinical problems. Though domain-specific fine-tuning can effectively inject medical knowledge into LLMs, it often leads to catastrophic forgetting of previously acquired knowledge and instruction following abilities. In this paper, we investigate catastrophic forgetting in medical knowledge injection and uncover a notable proximity-dependent forgetting nature: knowledge that is closely related to the injected knowledge is more likely to be forgotten, whereas distant knowledge is less affected. Moreover, we observe that existing mitigation techniques fail to effectively address this proximity-dependent forgetting. Inspired by this finding as well as human learning process, we propose a novel \textbf{Inter}nal K\textbf{n}owledge \textbf{A}ugmentation \textbf{L}earning (\textbf{InternAL}) approach that mitigates catastrophic forgetting by exploiting LLM's internal knowledge. Specifically, we first extract internal knowledge from the target LLM that is directly related to the injected knowledge, using questions generated from the injected knowledge. Then we augment the original knowledge injection dataset with the extracted internal knowledge, and train the LLM on this augmented dataset to enhance the model's ability to retain prior knowledge that is closely related to the injected knowledge. Experimental results on several representative LLMs (Llama, Qwen) show that our method significantly alleviates the forgetting of prior knowledge, especially the knowledge that is closely related to the injected knowledge. The datasets and codes will be released for future research.