Move-in-2D: 2D-Conditioned Human Motion Generation
Generating realistic human videos remains a challenging task, with the most
effective methods currently relying on a human motion sequence as a control
signal. Existing approaches often use existing motion extracted from other
videos, which restricts applications to specific motion types and global scene
matching. We propose Move-in-2D, a novel approach to generate human motion
sequences conditioned on a scene image, allowing for diverse motion that adapts
to different scenes. Our approach utilizes a diffusion model that accepts both
a scene image and text prompt as inputs, producing a motion sequence tailored
to the scene. To train this model, we collect a large-scale video dataset
featuring single-human activities, annotating each video with the corresponding
human motion as the target output. Experiments demonstrate that our method
effectively predicts human motion that aligns with the scene image after
projection. Furthermore, we show that the generated motion sequence improves
human motion quality in video synthesis tasks.
6762c4d82faaf11234a44936
Hsin-Ping Huang
claimed_verified
2024-12-19T18:02:01.287Z
667b2ee8e005e1dbcc76e2e2
/avatars/8b2f5f997f0ed5ae9f4f274941933c40.svg
Hsin-Ping Huang
user
hsinh
6762c4d82faaf11234a44937
Yang Zhou
6762c4d82faaf11234a44938
Jui-Hsien Wang
6762c4d82faaf11234a44939
Difan Liu
6762c4d82faaf11234a4493a
Feng Liu
6762c4d82faaf11234a4493b
Ming-Hsuan Yang
6762c4d82faaf11234a4493c
Zhan Xu
Move-in-2D: 2D-Conditioned Human Motion Generation
Generating realistic human videos remains a challenging task, with the most
effective methods currently relying on a human motion sequence as a control
signal. Existing approaches often use existing motion extracted from other
videos, which restricts applications to specific motion types and global scene
matching. We propose Move-in-2D, a novel approach to generate human motion
sequences conditioned on a scene image, allowing for diverse motion that adapts
to different scenes. Our approach utilizes a diffusion model that accepts both
a scene image and text prompt as inputs, producing a motion sequence tailored
to the scene. To train this model, we collect a large-scale video dataset
featuring single-human activities, annotating each video with the corresponding
human motion as the target output. Experiments demonstrate that our method
effectively predicts human motion that aligns with the scene image after
projection. Furthermore, we show that the generated motion sequence improves
human motion quality in video synthesis tasks.
A diffusion model generates human motion sequences conditioned on scene images and text prompts, showing improved quality in video synthesis.
diffusion model scene image text prompt human motion sequences video dataset single-human activities