Taming Modality Entanglement in Continual Audio-Visual Segmentation
Recently, significant progress has been made in multi-modal continual
learning, aiming to learn new tasks sequentially in multi-modal settings while
preserving performance on previously learned ones. However, existing methods
mainly focus on coarse-grained tasks, with limitations in addressing modality
entanglement in fine-grained continual learning settings. To bridge this gap,
we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to
continuously segment new classes guided by audio. Through comprehensive
analysis, two critical challenges are identified: 1) multi-modal semantic
drift, where a sounding objects is labeled as background in sequential tasks;
2) co-occurrence confusion, where frequent co-occurring classes tend to be
confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework
is designed to address these challenges. Specifically, for multi-modal semantic
drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select
samples with high modal consistency for rehearsal. Meanwhile, for co-occurence
confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed,
allowing for the increase of rehearsal sample frequency of those confusable
classes during training process. Moreover, we construct three audio-visual
incremental scenarios to verify effectiveness of our method. Comprehensive
experiments demonstrate that our method significantly outperforms single-modal
continual learning methods.
68f9ab58b9b2e4ae046738b1
Yuyang Hong
claimed_verified
2025-10-25T14:34:02.018Z
66e03d7123e5f162e7e5c5ac
/avatars/28089b6ffeacafff2c798b7579689c1e.svg
hongyuyang
user
hongyuyang23casia
68f9ab58b9b2e4ae046738b2
Qi Yang
68f9ab58b9b2e4ae046738b3
Tao Zhang
68f9ab58b9b2e4ae046738b4
Zili Wang
68f9ab58b9b2e4ae046738b5
Zhaojin Fu
68f9ab58b9b2e4ae046738b6
Kun Ding
68f9ab58b9b2e4ae046738b7
Bin Fan
68f9ab58b9b2e4ae046738b8
Shiming Xiang
Taming Modality Entanglement in Continual Audio-Visual Segmentation
Recently, significant progress has been made in multi-modal continual
learning, aiming to learn new tasks sequentially in multi-modal settings while
preserving performance on previously learned ones. However, existing methods
mainly focus on coarse-grained tasks, with limitations in addressing modality
entanglement in fine-grained continual learning settings. To bridge this gap,
we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to
continuously segment new classes guided by audio. Through comprehensive
analysis, two critical challenges are identified: 1) multi-modal semantic
drift, where a sounding objects is labeled as background in sequential tasks;
2) co-occurrence confusion, where frequent co-occurring classes tend to be
confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework
is designed to address these challenges. Specifically, for multi-modal semantic
drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select
samples with high modal consistency for rehearsal. Meanwhile, for co-occurence
confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed,
allowing for the increase of rehearsal sample frequency of those confusable
classes during training process. Moreover, we construct three audio-visual
incremental scenarios to verify effectiveness of our method. Comprehensive
experiments demonstrate that our method significantly outperforms single-modal
continual learning methods.
A novel Continual Audio-Visual Segmentation (CAVS) task addresses challenges in multi-modal continual learning through a Collision-based Multi-modal Rehearsal (CMR) framework, improving performance over single-modal methods.
multi-modal continual learning Continual Audio-Visual Segmentation CAVS multi-modal semantic drift co-occurrence confusion Collision-based Multi-modal Rehearsal CMR Multi-modal Sample Selection MSS Collision-based Sample Rehearsal CSR