Controllable Human-Object Interaction Synthesis
Synthesizing semantic-aware, long-horizon, human-object interaction is
critical to simulate realistic human behaviors. In this work, we address the
challenging problem of generating synchronized object motion and human motion
guided by language descriptions in 3D scenes. We propose Controllable
Human-Object Interaction Synthesis (CHOIS), an approach that generates object
motion and human motion simultaneously using a conditional diffusion model
given a language description, initial object and human states, and sparse
object waypoints. While language descriptions inform style and intent,
waypoints ground the motion in the scene and can be effectively extracted using
high-level planning methods. Naively applying a diffusion model fails to
predict object motion aligned with the input waypoints and cannot ensure the
realism of interactions that require precise hand-object contact and
appropriate contact grounded by the floor. To overcome these problems, we
introduce an object geometry loss as additional supervision to improve the
matching between generated object motion and input object waypoints. In
addition, we design guidance terms to enforce contact constraints during the
sampling process of the trained diffusion model.
65729540cca2247f3f2c59f4
Jiaman Li
admin_assigned
2023-12-08T10:27:54.335Z
653b7d1732bd4db35d9ada4c
/avatars/b829135b7b36c07e5f47c343f8b82594.svg
Jiaman Li
user
jiaman
65729540cca2247f3f2c59f5
Alexander Clegg
admin_assigned
2023-12-08T10:28:02.186Z
63eabb40a332618465d4f698
https://cdn-avatars.huggingface.co/v1/production/uploads/1676327808512-63eabb40a332618465d4f698.jpeg
Alexander Clegg
user
aclegg3
65729540cca2247f3f2c59f6
Roozbeh Mottaghi
65729540cca2247f3f2c59f7
Jiajun Wu
admin_assigned
2023-12-08T10:28:27.192Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
65729540cca2247f3f2c59f8
Xavier Puig
admin_assigned
2023-12-08T10:28:33.829Z
64669f6afc452cf123c0a0b6
/avatars/9dec7a63d38bfd8606a468b0eb77c6e5.svg
Xavier Puig
user
xavierpuigf
65729540cca2247f3f2c59f9
C. Karen Liu
Controllable Human-Object Interaction Synthesis
Synthesizing semantic-aware, long-horizon, human-object interaction is
critical to simulate realistic human behaviors. In this work, we address the
challenging problem of generating synchronized object motion and human motion
guided by language descriptions in 3D scenes. We propose Controllable
Human-Object Interaction Synthesis (CHOIS), an approach that generates object
motion and human motion simultaneously using a conditional diffusion model
given a language description, initial object and human states, and sparse
object waypoints. While language descriptions inform style and intent,
waypoints ground the motion in the scene and can be effectively extracted using
high-level planning methods. Naively applying a diffusion model fails to
predict object motion aligned with the input waypoints and cannot ensure the
realism of interactions that require precise hand-object contact and
appropriate contact grounded by the floor. To overcome these problems, we
introduce an object geometry loss as additional supervision to improve the
matching between generated object motion and input object waypoints. In
addition, we design guidance terms to enforce contact constraints during the
sampling process of the trained diffusion model.
CHOIS uses a conditional diffusion model to generate synchronized human and object motion in 3D scenes, guided by language descriptions and waypoints, with improvements for object geometry alignment and contact constraints.
conditional diffusion model object motion human motion language descriptions initial object and human states sparse object waypoints object geometry loss guidance terms contact constraints