MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Large language models (LLMs) have recently shown strong potential in
audio-visual speech recognition (AVSR), but their high computational demands
and sensitivity to token granularity limit their practicality in
resource-constrained settings. Token compression methods can reduce inference
cost, but they require fixing a compression rate in advance and produce a
single fixed-length output, offering no flexibility to balance information
density and efficiency at inference time. Matryoshka representation learning
(MRL) addresses this by enabling a single model to operate across multiple
token granularities, allowing compression rates to be adjusted dynamically.
However, current MRL-based methods treat each scale independently during
training, limiting cross-scale generalization, robustness at high compression,
and interpretability. To overcome these limitations, we propose MoME (Mixture
of Matryoshka Experts), a novel framework that integrates sparse
Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen
LLM with top-k routed and shared experts, allowing dynamic capacity allocation
across scales and modalities. A shared router promotes consistent expert
activation across granularities, enabling compressed sequences to benefit from
representations learned at lower compression. Experiments on LRS2 and LRS3
demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR,
and VSR tasks, while requiring significantly fewer parameters and maintaining
robustness under noise. MoME unifies the adaptability of MRL with the
efficiency of MoE, offering a scalable and interpretable solution for
resource-aware speech recognition.
68e4bc16e4e093a7044e4e34
Umberto Cappellazzo
claimed_verified
2025-10-07T12:26:03.309Z
64903f017b630c141867877f
/avatars/3c7b248baed446ecc2b6adc2c444320d.svg
Umberto Cappellazzo
user
hisoka94
68e4bc16e4e093a7044e4e35
Minsu Kim
68e4bc16e4e093a7044e4e36
Pingchuan Ma
68e4bc16e4e093a7044e4e37
Honglie Chen
68e4bc16e4e093a7044e4e38
Xubo Liu
68e4bc16e4e093a7044e4e39
Stavros Petridis
68e4bc16e4e093a7044e4e3a
Maja Pantic
MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Large language models (LLMs) have recently shown strong potential in
audio-visual speech recognition (AVSR), but their high computational demands
and sensitivity to token granularity limit their practicality in
resource-constrained settings. Token compression methods can reduce inference
cost, but they require fixing a compression rate in advance and produce a
single fixed-length output, offering no flexibility to balance information
density and efficiency at inference time. Matryoshka representation learning
(MRL) addresses this by enabling a single model to operate across multiple
token granularities, allowing compression rates to be adjusted dynamically.
However, current MRL-based methods treat each scale independently during
training, limiting cross-scale generalization, robustness at high compression,
and interpretability. To overcome these limitations, we propose MoME (Mixture
of Matryoshka Experts), a novel framework that integrates sparse
Mixture-of-Experts (MoE) into MRL-based LLMs for AVSR. MoME augments a frozen
LLM with top-k routed and shared experts, allowing dynamic capacity allocation
across scales and modalities. A shared router promotes consistent expert
activation across granularities, enabling compressed sequences to benefit from
representations learned at lower compression. Experiments on LRS2 and LRS3
demonstrate that MoME achieves state-of-the-art performance across AVSR, ASR,
and VSR tasks, while requiring significantly fewer parameters and maintaining
robustness under noise. MoME unifies the adaptability of MRL with the
efficiency of MoE, offering a scalable and interpretable solution for
resource-aware speech recognition.
MoME, a novel framework integrating sparse Mixture-of-Experts into Matryoshka representation learning, enhances audio-visual speech recognition by dynamically adjusting capacity across scales and modalities, achieving state-of-the-art performance with fewer parameters.
large language models audio-visual speech recognition token compression Matryoshka representation learning Mixture-of-Experts MoME LRS2 LRS3 ASR VSR parameter-efficient robustness noise