UniMIC: Token-Based Multimodal Interactive Coding for Human-AI
  Collaboration
The rapid progress of Large Multimodal Models (LMMs) and cloud-based AI
agents is transforming human-AI collaboration into bidirectional, multimodal
interaction. However, existing codecs remain optimized for unimodal, one-way
communication, resulting in repeated degradation under conventional
compress-transmit-reconstruct pipelines. To address this limitation, we propose
UniMIC, a Unified token-based Multimodal Interactive Coding framework that
bridges edge devices and cloud AI agents. Instead of transmitting raw pixels or
plain text, UniMIC employs compact tokenized representations as the
communication medium, enabling efficient low-bitrate transmission while
maintaining compatibility with LMMs. To further enhance compression,
lightweight Transformer-based entropy models with scenario-specific
designs-generic, masked, and text-conditioned-effectively minimize inter-token
redundancy. Extensive experiments on text-to-image generation, text-guided
inpainting, outpainting, and visual question answering show that UniMIC
achieves substantial bitrate savings and remains robust even at ultra-low
bitrates (<0.05bpp), without compromising downstream task performance. These
results establish UniMIC as a practical and forward-looking paradigm for
next-generation multimodal interactive communication.
68da9e53d2bf1f4b15ec7151
Qi Mao
claimed_verified
2025-09-30T06:35:51.051Z
6388a7e98a5dbe2f3dc61faa
https://cdn-avatars.huggingface.co/v1/production/uploads/6388a7e98a5dbe2f3dc61faa/Zj8pR7BG_wOQjsYzosb5p.jpeg
Qi Mao
user
HelenMao
68da9e53d2bf1f4b15ec7152
Tinghan Yang
68da9e53d2bf1f4b15ec7153
Jiahao Li
68da9e53d2bf1f4b15ec7154
Bin Li
68da9e53d2bf1f4b15ec7155
Libiao Jin
68da9e53d2bf1f4b15ec7156
Yan Lu
UniMIC: Token-Based Multimodal Interactive Coding for Human-AI
  Collaboration
The rapid progress of Large Multimodal Models (LMMs) and cloud-based AI
agents is transforming human-AI collaboration into bidirectional, multimodal
interaction. However, existing codecs remain optimized for unimodal, one-way
communication, resulting in repeated degradation under conventional
compress-transmit-reconstruct pipelines. To address this limitation, we propose
UniMIC, a Unified token-based Multimodal Interactive Coding framework that
bridges edge devices and cloud AI agents. Instead of transmitting raw pixels or
plain text, UniMIC employs compact tokenized representations as the
communication medium, enabling efficient low-bitrate transmission while
maintaining compatibility with LMMs. To further enhance compression,
lightweight Transformer-based entropy models with scenario-specific
designs-generic, masked, and text-conditioned-effectively minimize inter-token
redundancy. Extensive experiments on text-to-image generation, text-guided
inpainting, outpainting, and visual question answering show that UniMIC
achieves substantial bitrate savings and remains robust even at ultra-low
bitrates (<0.05bpp), without compromising downstream task performance. These
results establish UniMIC as a practical and forward-looking paradigm for
next-generation multimodal interactive communication.
UniMIC, a unified token-based framework, enhances multimodal communication by using compact tokenized representations and lightweight Transformer-based entropy models, achieving significant bitrate savings without compromising performance.
Large Multimodal Models UniMIC token-based multimodal interactive coding edge devices cloud AI agents compact tokenized representations lightweight Transformer-based entropy models text-to-image generation text-guided inpainting outpainting visual question answering bitrate savings ultra-low bitrates