In-2-4D: Inbetweening from Two Single-View Images to 4D Generation
We propose a new problem, In-2-4D, for generative 4D (i.e., 3D + motion)
inbetweening from a minimalistic input setting: two single-view images
capturing an object in two distinct motion states. Given two images
representing the start and end states of an object in motion, our goal is to
generate and reconstruct the motion in 4D. We utilize a video interpolation
model to predict the motion, but large frame-to-frame motions can lead to
ambiguous interpretations. To overcome this, we employ a hierarchical approach
to identify keyframes that are visually close to the input states and show
significant motion, then generate smooth fragments between them. For each
fragment, we construct the 3D representation of the keyframe using Gaussian
Splatting. The temporal frames within the fragment guide the motion, enabling
their transformation into dynamic Gaussians through a deformation field. To
improve temporal consistency and refine 3D motion, we expand the self-attention
of multi-view diffusion across timesteps and apply rigid transformation
regularization. Finally, we merge the independently generated 3D motion
segments by interpolating boundary deformation fields and optimizing them to
align with the guiding video, ensuring smooth and flicker-free transitions.
Through extensive qualitative and quantitiave experiments as well as a user
study, we show the effectiveness of our method and its components. The project
page is available at https://in-2-4d.github.io/
67fc6d8d74c3c0f0d6f24c3b
Sauradip Nag
67fc6d8d74c3c0f0d6f24c3c
Daniel Cohen-Or
67fc6d8d74c3c0f0d6f24c3d
Hao Zhang
67fc6d8d74c3c0f0d6f24c3e
Ali Mahdavi-Amiri
In-2-4D: Inbetweening from Two Single-View Images to 4D Generation
We propose a new problem, In-2-4D, for generative 4D (i.e., 3D + motion)
inbetweening from a minimalistic input setting: two single-view images
capturing an object in two distinct motion states. Given two images
representing the start and end states of an object in motion, our goal is to
generate and reconstruct the motion in 4D. We utilize a video interpolation
model to predict the motion, but large frame-to-frame motions can lead to
ambiguous interpretations. To overcome this, we employ a hierarchical approach
to identify keyframes that are visually close to the input states and show
significant motion, then generate smooth fragments between them. For each
fragment, we construct the 3D representation of the keyframe using Gaussian
Splatting. The temporal frames within the fragment guide the motion, enabling
their transformation into dynamic Gaussians through a deformation field. To
improve temporal consistency and refine 3D motion, we expand the self-attention
of multi-view diffusion across timesteps and apply rigid transformation
regularization. Finally, we merge the independently generated 3D motion
segments by interpolating boundary deformation fields and optimizing them to
align with the guiding video, ensuring smooth and flicker-free transitions.
Through extensive qualitative and quantitiave experiments as well as a user
study, we show the effectiveness of our method and its components. The project
page is available at https://in-2-4d.github.io/
A hierarchical 4D inbetweening approach generates smooth motion from two input images using Gaussian Splatting, multi-view diffusion, and self-attention across timesteps.
Gaussian Splatting multi-view diffusion self-attention deformation field rigid transformation regularization