Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video
  Generation
We present Stable Part Diffusion 4D (SP4D), a framework for generating paired
RGB and kinematic part videos from monocular inputs. Unlike conventional part
segmentation methods that rely on appearance-based semantic cues, SP4D learns
to produce kinematic parts - structural components aligned with object
articulation and consistent across views and time. SP4D adopts a dual-branch
diffusion model that jointly synthesizes RGB frames and corresponding part
segmentation maps. To simplify the architecture and flexibly enable different
part counts, we introduce a spatial color encoding scheme that maps part masks
to continuous RGB-like images. This encoding allows the segmentation branch to
share the latent VAE from the RGB branch, while enabling part segmentation to
be recovered via straightforward post-processing. A Bidirectional Diffusion
Fusion (BiDiFuse) module enhances cross-branch consistency, supported by a
contrastive part consistency loss to promote spatial and temporal alignment of
part predictions. We demonstrate that the generated 2D part maps can be lifted
to 3D to derive skeletal structures and harmonic skinning weights with few
manual adjustments. To train and evaluate SP4D, we construct KinematicParts20K,
a curated dataset of over 20K rigged objects selected and processed from
Objaverse XL (Deitke et al., 2023), each paired with multi-view RGB and part
video sequences. Experiments show that SP4D generalizes strongly to diverse
scenarios, including real-world videos, novel generated objects, and rare
articulated poses, producing kinematic-aware outputs suitable for downstream
animation and motion-related tasks.
68caf3075a7803ff3be42c6f
Hao Zhang
68caf3075a7803ff3be42c70
Chun-Han Yao
68caf3075a7803ff3be42c71
Simon Donné
68caf3075a7803ff3be42c72
Narendra Ahuja
68caf3075a7803ff3be42c73
Varun Jampani
Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video
  Generation
We present Stable Part Diffusion 4D (SP4D), a framework for generating paired
RGB and kinematic part videos from monocular inputs. Unlike conventional part
segmentation methods that rely on appearance-based semantic cues, SP4D learns
to produce kinematic parts - structural components aligned with object
articulation and consistent across views and time. SP4D adopts a dual-branch
diffusion model that jointly synthesizes RGB frames and corresponding part
segmentation maps. To simplify the architecture and flexibly enable different
part counts, we introduce a spatial color encoding scheme that maps part masks
to continuous RGB-like images. This encoding allows the segmentation branch to
share the latent VAE from the RGB branch, while enabling part segmentation to
be recovered via straightforward post-processing. A Bidirectional Diffusion
Fusion (BiDiFuse) module enhances cross-branch consistency, supported by a
contrastive part consistency loss to promote spatial and temporal alignment of
part predictions. We demonstrate that the generated 2D part maps can be lifted
to 3D to derive skeletal structures and harmonic skinning weights with few
manual adjustments. To train and evaluate SP4D, we construct KinematicParts20K,
a curated dataset of over 20K rigged objects selected and processed from
Objaverse XL (Deitke et al., 2023), each paired with multi-view RGB and part
video sequences. Experiments show that SP4D generalizes strongly to diverse
scenarios, including real-world videos, novel generated objects, and rare
articulated poses, producing kinematic-aware outputs suitable for downstream
animation and motion-related tasks.
SP4D generates paired RGB and kinematic part videos from monocular inputs using a dual-branch diffusion model with spatial color encoding and BiDiFuse module, demonstrating strong generalization to diverse scenarios.
dual-branch diffusion model spatial color encoding latent VAE Bidirectional Diffusion Fusion (BiDiFuse) contrastive part consistency loss skeletal structures harmonic skinning weights KinematicParts20K Objaverse XL