Efficient Multi-modal Large Language Models via Progressive Consistency
  Distillation
Visual tokens consume substantial computational resources in multi-modal
large models (MLLMs), significantly compromising their efficiency. Recent works
have attempted to improve efficiency by compressing visual tokens during
training, either through modifications to model components or by introducing
additional parameters. However, they often overlook the increased learning
difficulty caused by such compression, as the model's parameter space struggles
to quickly adapt to the substantial perturbations in the feature space induced
by token compression. In this work, we propose to develop Efficient MLLMs via
Progressive Consistency Distillation (EPIC), a progressive learning framework.
Specifically, by decomposing the feature space perturbations introduced by
token compression along the token-wise and layer-wise dimensions, we introduce
token consistency distillation and layer consistency distillation,
respectively, aiming to reduce the training difficulty by leveraging guidance
from a teacher model and following a progressive learning trajectory. Extensive
experiments demonstrate the superior effectiveness, robustness, and
generalization capabilities of our proposed framework.
68e34f7173e20ab5778420bb
Zichen Wen
claimed_verified
2025-10-10T09:39:01.706Z
653b8c3e97a4d71d950e2f20
/avatars/b68880022e14556d0be58c69615db3be.svg
Zichen Wen
user
zichenwen
68e34f7173e20ab5778420bc
Shaobo Wang
claimed_verified
2025-10-14T07:32:40.937Z
66968099c952e09a4cb29f78
https://cdn-avatars.huggingface.co/v1/production/uploads/66968099c952e09a4cb29f78/n90NI2R3E9_RqCyMjDCQF.webp
Wang
user
Steven-Shaobo
68e34f7173e20ab5778420bd
Yufa Zhou
claimed_verified
2025-10-16T06:42:26.827Z
658ab894c4b2004663dff3ae
https://cdn-avatars.huggingface.co/v1/production/uploads/658ab894c4b2004663dff3ae/oPRnFuW2Imaa2KkYWNbSf.jpeg
YUFA ZHOU
user
MasterZhou
68e34f7173e20ab5778420be
Junyuan Zhang
68e34f7173e20ab5778420bf
Qintong Zhang
68e34f7173e20ab5778420c0
Yifeng Gao
68e34f7173e20ab5778420c1
Zhaorun Chen
68e34f7173e20ab5778420c2
Bin Wang
68e34f7173e20ab5778420c3
Weijia Li
68e34f7173e20ab5778420c4
Conghui He
68e34f7173e20ab5778420c5
Linfeng Zhang
Efficient Multi-modal Large Language Models via Progressive Consistency
  Distillation
Visual tokens consume substantial computational resources in multi-modal
large models (MLLMs), significantly compromising their efficiency. Recent works
have attempted to improve efficiency by compressing visual tokens during
training, either through modifications to model components or by introducing
additional parameters. However, they often overlook the increased learning
difficulty caused by such compression, as the model's parameter space struggles
to quickly adapt to the substantial perturbations in the feature space induced
by token compression. In this work, we propose to develop Efficient MLLMs via
Progressive Consistency Distillation (EPIC), a progressive learning framework.
Specifically, by decomposing the feature space perturbations introduced by
token compression along the token-wise and layer-wise dimensions, we introduce
token consistency distillation and layer consistency distillation,
respectively, aiming to reduce the training difficulty by leveraging guidance
from a teacher model and following a progressive learning trajectory. Extensive
experiments demonstrate the superior effectiveness, robustness, and
generalization capabilities of our proposed framework.
EPIC, a progressive learning framework, improves the efficiency of multi-modal large models by reducing training difficulty through token and layer consistency distillation during visual token compression.
visual tokens multi-modal large models MLLMs token compression parameter space feature space progressive learning framework token consistency distillation layer consistency distillation teacher model progressive learning trajectory