R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
In large multimodal models (LMMs), the perception of non-language modalities
(e.g., visual representations) is usually not on par with the large language
models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on
challenging downstream tasks. This weakness has been recently mitigated by
replacing the vision encoder with a mixture-of-experts (MoE), which provides
rich, multi-granularity, and diverse representations required by diverse
downstream tasks. The performance of multimodal MoE largely depends on its
router, which reweights and mixes the representations of different experts for
each input. However, we find that the end-to-end trained router does not always
produce the optimal routing weights for every test sample. To bridge the gap,
we propose a novel and efficient method "Re-Routing in Test-Time(R2-T2) that
locally optimizes the vector of routing weights in test-time by moving it
toward those vectors of the correctly predicted samples in a neighborhood of
the test sample. We propose three R2-T2 strategies with different optimization
objectives and neighbor-search spaces. R2-T2 consistently and greatly improves
state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks,
without training any base-model parameters.
67c12b5def9af74902537b98
Zhongyang Li
claimed_verified
2025-02-28T12:14:22.809Z
671002fd13203512e7b8f9e3
/avatars/313d8ea313ed300750cfdaaca44fdb6e.svg
Zhongyang Li
user
Lzy01241010
67c12b5def9af74902537b99
Ziyue Li
67c12b5def9af74902537b9a
Tianyi Zhou
claimed_verified
2025-02-28T12:14:16.482Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
In large multimodal models (LMMs), the perception of non-language modalities
(e.g., visual representations) is usually not on par with the large language
models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on
challenging downstream tasks. This weakness has been recently mitigated by
replacing the vision encoder with a mixture-of-experts (MoE), which provides
rich, multi-granularity, and diverse representations required by diverse
downstream tasks. The performance of multimodal MoE largely depends on its
router, which reweights and mixes the representations of different experts for
each input. However, we find that the end-to-end trained router does not always
produce the optimal routing weights for every test sample. To bridge the gap,
we propose a novel and efficient method "Re-Routing in Test-Time(R2-T2) that
locally optimizes the vector of routing weights in test-time by moving it
toward those vectors of the correctly predicted samples in a neighborhood of
the test sample. We propose three R2-T2 strategies with different optimization
objectives and neighbor-search spaces. R2-T2 consistently and greatly improves
state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks,
without training any base-model parameters.
A test-time re-routing method improves the performance of multimodal models by optimizing routing weights for non-language modalities, enhancing accuracy in challenging tasks without training.
multimodal models large language models vision encoder mixture-of-experts router test-time re-routing R2-T2 routing weights end-to-end training optimization strategies benchmark tasks