Multi-Modal Human Motion Alignment for Cross-Modal Retrieval, Recognition, and Generation
Kaleab Kinfu, Rene Vidal
Recent advances in multi-modal representation learning have led to unified embedding spaces that align modalities such as images, text, audio, and vision. However, human motion sequences, a modality that is fundamental to understanding dynamic human activities, remains largely unrepresented in these frameworks. Semantic understanding of actions requires multi-modal grounding: text conveys descriptive semantics, vision provides visual context, and audio provides environmental cues. To bridge this gap, we extend the LanguageBind embedding space to incorporate human motion. Specifically, we introduce Multi-Scale Temporal Motion Transformer (MuTMoT), a transformer-based encoder that maps motion sequences into semantically meaningful embeddings. Alignment is achieved via diverse cross-modal supervision: motion-text pairs from HumanML3D and KIT-ML, motion-video pairs rendered from AMASS, and motion-video-audio triplets from AIST++. Beyond alignment, we introduce REALM, a retrieval-augmented latent diffusion model for motion synthesis. Our approach achieves state-of-the-art or competitive performance across motion reconstruction, cross-modal retrieval, zero-shot action recognition, and text-to-motion generation benchmarks.