KORE: Enhancing Knowledge Injection for Large Multimodal Models via
  Knowledge-Oriented Augmentations and Constraints
Large Multimodal Models encode extensive factual knowledge in their
pre-trained weights. However, its knowledge remains static and limited, unable
to keep pace with real-world developments, which hinders continuous knowledge
acquisition. Effective knowledge injection thus becomes critical, involving two
goals: knowledge adaptation (injecting new knowledge) and knowledge retention
(preserving old knowledge). Existing methods often struggle to learn new
knowledge and suffer from catastrophic forgetting. To address this, we propose
KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints
for injecting new knowledge into large multimodal models while preserving old
knowledge. Unlike general text or image data augmentation, KORE automatically
converts individual knowledge items into structured and comprehensive knowledge
to ensure that the model accurately learns new knowledge, enabling accurate
adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix
of LMM's linear layer activations and initializes the adapter by projecting the
original weights into the matrix's null space, defining a fine-tuning direction
that minimizes interference with previous knowledge, enabling powerful
retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B,
LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new
knowledge injection performance and effectively mitigates catastrophic
forgetting.
68f98c87b9b2e4ae04673796
Kailin Jiang
claimed_verified
2025-10-23T14:50:15.872Z
65745569839aa08899ea5d27
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg
kailinjiang
user
kailinjiang
68f98c87b9b2e4ae04673797
Hongbo Jiang
68f98c87b9b2e4ae04673798
Ning Jiang
68f98c87b9b2e4ae04673799
Zhi Gao
68f98c87b9b2e4ae0467379a
Jinhe Bi
68f98c87b9b2e4ae0467379b
Yuchen Ren
68f98c87b9b2e4ae0467379c
Bin Li
68f98c87b9b2e4ae0467379d
Yuntao Du
68f98c87b9b2e4ae0467379e
Lei Liu
68f98c87b9b2e4ae0467379f
Qing Li
KORE: Enhancing Knowledge Injection for Large Multimodal Models via
  Knowledge-Oriented Augmentations and Constraints
Large Multimodal Models encode extensive factual knowledge in their
pre-trained weights. However, its knowledge remains static and limited, unable
to keep pace with real-world developments, which hinders continuous knowledge
acquisition. Effective knowledge injection thus becomes critical, involving two
goals: knowledge adaptation (injecting new knowledge) and knowledge retention
(preserving old knowledge). Existing methods often struggle to learn new
knowledge and suffer from catastrophic forgetting. To address this, we propose
KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints
for injecting new knowledge into large multimodal models while preserving old
knowledge. Unlike general text or image data augmentation, KORE automatically
converts individual knowledge items into structured and comprehensive knowledge
to ensure that the model accurately learns new knowledge, enabling accurate
adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix
of LMM's linear layer activations and initializes the adapter by projecting the
original weights into the matrix's null space, defining a fine-tuning direction
that minimizes interference with previous knowledge, enabling powerful
retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B,
LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new
knowledge injection performance and effectively mitigates catastrophic
forgetting.
KORE is a method for injecting new knowledge into large multimodal models while preserving old knowledge, using structured augmentations and covariance matrix constraints to minimize catastrophic forgetting.
knowledge adaptation knowledge retention catastrophic forgetting KORE KnOwledge-oRientEd augmentations covariance matrix linear layer activations adapter fine-tuning direction LMMs LLaVA-v1.5-7B LLaVA-v1.5-13B Qwen2.5-VL-7B