Is Extending Modality The Right Path Towards Omni-Modality?
Omni-modal language models (OLMs) aim to integrate and reason over diverse
input modalities--such as text, images, video, and audio--while maintaining
strong language capabilities. Despite recent advancements, existing models,
especially open-source ones, remain far from true omni-modality, struggling to
generalize beyond the specific modality pairs they are trained on or to achieve
strong performance when processing multi-modal inputs. We study the effect of
extending modality, the dominant technique for training multimodal models,
where an off-the-shelf language model is fine-tuned on target-domain and
language data. Specifically, we investigate three key questions: (1) Does
modality extension compromise core language abilities? (2) Can model merging
effectively integrate independently fine-tuned modality-specific models to
achieve omni-modality? (3) Does omni-modality extension lead to better
knowledge sharing and generalization compared to sequential extension? Through
extensive experiments, we analyze these trade-offs and provide insights into
the feasibility of achieving true omni-modality using current approaches.
683e77d41417d107337abf6e
Tinghui Zhu
claimed_verified
2025-06-09T10:12:47.572Z
643f9e2288d9d4488fd81c52
/avatars/e589c9cbd47022883cf33d7555bee89c.svg
Tinghui Zhu
user
DarthZhu
683e77d41417d107337abf6f
Kai Zhang
683e77d41417d107337abf70
Muhao Chen
683e77d41417d107337abf71
Yu Su
Is Extending Modality The Right Path Towards Omni-Modality?
Omni-modal language models (OLMs) aim to integrate and reason over diverse
input modalities--such as text, images, video, and audio--while maintaining
strong language capabilities. Despite recent advancements, existing models,
especially open-source ones, remain far from true omni-modality, struggling to
generalize beyond the specific modality pairs they are trained on or to achieve
strong performance when processing multi-modal inputs. We study the effect of
extending modality, the dominant technique for training multimodal models,
where an off-the-shelf language model is fine-tuned on target-domain and
language data. Specifically, we investigate three key questions: (1) Does
modality extension compromise core language abilities? (2) Can model merging
effectively integrate independently fine-tuned modality-specific models to
achieve omni-modality? (3) Does omni-modality extension lead to better
knowledge sharing and generalization compared to sequential extension? Through
extensive experiments, we analyze these trade-offs and provide insights into
the feasibility of achieving true omni-modality using current approaches.
Research investigates the impact of extending modality and model merging on maintaining language abilities and generalization in omni-modal language models.
omni-modal language models modality extension fine-tuning language abilities model merging generalization true omni-modality