MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
Image-to-video generation has made remarkable progress with the advancements
in diffusion models, yet generating videos with realistic motion remains highly
challenging. This difficulty arises from the complexity of accurately modeling
motion, which involves capturing physical constraints, object interactions, and
domain-specific dynamics that are not easily generalized across diverse
scenarios. To address this, we propose MotionRAG, a retrieval-augmented
framework that enhances motion realism by adapting motion priors from relevant
reference videos through Context-Aware Motion Adaptation (CAMA). The key
technical innovations include: (i) a retrieval-based pipeline extracting
high-level motion features using video encoder and specialized resamplers to
distill semantic motion representations; (ii) an in-context learning approach
for motion adaptation implemented through a causal transformer architecture;
(iii) an attention-based motion injection adapter that seamlessly integrates
transferred motion features into pretrained video diffusion models. Extensive
experiments demonstrate that our method achieves significant improvements
across multiple domains and various base models, all with negligible
computational overhead during inference. Furthermore, our modular design
enables zero-shot generalization to new domains by simply updating the
retrieval database without retraining any components. This research enhances
the core capability of video generation systems by enabling the effective
retrieval and transfer of motion priors, facilitating the synthesis of
realistic motion dynamics.
68dc90ca4159d1f2418f9ab8
Chenhui Zhu
claimed_verified
2025-10-01T10:21:56.849Z
65e55ca4a0681de63022843e
/avatars/5b1ac4a81f0c38fda6f47b392f7474c8.svg
zhu chenhui
user
flateon
68dc90ca4159d1f2418f9ab9
Yilu Wu
68dc90ca4159d1f2418f9aba
Shuai Wang
claimed_verified
2025-10-01T10:21:54.463Z
66615c855fd9d736e670e0a9
/avatars/0ff3127b513552432a7c651e21d7f283.svg
wangshuai
user
wangsssssss
68dc90ca4159d1f2418f9abb
Gangshan Wu
68dc90ca4159d1f2418f9abc
Limin Wang
MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
Image-to-video generation has made remarkable progress with the advancements
in diffusion models, yet generating videos with realistic motion remains highly
challenging. This difficulty arises from the complexity of accurately modeling
motion, which involves capturing physical constraints, object interactions, and
domain-specific dynamics that are not easily generalized across diverse
scenarios. To address this, we propose MotionRAG, a retrieval-augmented
framework that enhances motion realism by adapting motion priors from relevant
reference videos through Context-Aware Motion Adaptation (CAMA). The key
technical innovations include: (i) a retrieval-based pipeline extracting
high-level motion features using video encoder and specialized resamplers to
distill semantic motion representations; (ii) an in-context learning approach
for motion adaptation implemented through a causal transformer architecture;
(iii) an attention-based motion injection adapter that seamlessly integrates
transferred motion features into pretrained video diffusion models. Extensive
experiments demonstrate that our method achieves significant improvements
across multiple domains and various base models, all with negligible
computational overhead during inference. Furthermore, our modular design
enables zero-shot generalization to new domains by simply updating the
retrieval database without retraining any components. This research enhances
the core capability of video generation systems by enabling the effective
retrieval and transfer of motion priors, facilitating the synthesis of
realistic motion dynamics.
MotionRAG enhances video generation by integrating motion priors from reference videos using a retrieval-augmented framework, improving motion realism with negligible computational overhead.
diffusion models MotionRAG Context-Aware Motion Adaptation (CAMA) video encoder specialized resamplers causal transformer architecture attention-based motion injection adapter zero-shot generalization