Boximator: Generating Rich and Controllable Motions for Video Synthesis
Generating rich and controllable motion is a pivotal challenge in video
synthesis. We propose Boximator, a new approach for fine-grained motion
control. Boximator introduces two constraint types: hard box and soft box.
Users select objects in the conditional frame using hard boxes and then use
either type of boxes to roughly or rigorously define the object's position,
shape, or motion path in future frames. Boximator functions as a plug-in for
existing video diffusion models. Its training process preserves the base
model's knowledge by freezing the original weights and training only the
control module. To address training challenges, we introduce a novel
self-tracking technique that greatly simplifies the learning of box-object
correlations. Empirically, Boximator achieves state-of-the-art video quality
(FVD) scores, improving on two base models, and further enhanced after
incorporating box constraints. Its robust motion controllability is validated
by drastic increases in the bounding box alignment metric. Human evaluation
also shows that users favor Boximator generation results over the base model.
65c06e55987c4be8a5379348
Jiawei Wang
claimed_verified
2025-01-15T21:21:29.998Z
6788122301d5ba1d3eff23ba
/avatars/203dc8e1d542be55ea16eafcd7f396ff.svg
Jiawei Wang
user
0nejiawei
65c06e55987c4be8a5379349
Yuchen Zhang
65c06e55987c4be8a537934a
Jiaxin Zou
admin_assigned
2024-02-05T12:15:20.773Z
655c485e71ed242b42701fb0
https://cdn-avatars.huggingface.co/v1/production/uploads/655c485e71ed242b42701fb0/7d6lHB-wMy7xNj3dWIp5y.jpeg
Jiaxin Zou
user
shayeboshi
65c06e55987c4be8a537934b
Yan Zeng
admin_assigned
2024-02-05T12:15:54.454Z
640ef2b2f2d7c41a1e9b6013
/avatars/556263c07df93942c22cda64e65b2e41.svg
Yan Zeng
user
zengyan-97
65c06e55987c4be8a537934c
Guoqiang Wei
admin_assigned
2024-02-05T12:16:00.898Z
63160a1d29411a6864b0e585
https://cdn-avatars.huggingface.co/v1/production/uploads/1668083049513-63160a1d29411a6864b0e585.jpeg
Guoqiang Wei
user
weigq
65c06e55987c4be8a537934d
Liping Yuan
65c06e55987c4be8a537934e
Hang Li
Boximator: Generating Rich and Controllable Motions for Video Synthesis
Generating rich and controllable motion is a pivotal challenge in video
synthesis. We propose Boximator, a new approach for fine-grained motion
control. Boximator introduces two constraint types: hard box and soft box.
Users select objects in the conditional frame using hard boxes and then use
either type of boxes to roughly or rigorously define the object's position,
shape, or motion path in future frames. Boximator functions as a plug-in for
existing video diffusion models. Its training process preserves the base
model's knowledge by freezing the original weights and training only the
control module. To address training challenges, we introduce a novel
self-tracking technique that greatly simplifies the learning of box-object
correlations. Empirically, Boximator achieves state-of-the-art video quality
(FVD) scores, improving on two base models, and further enhanced after
incorporating box constraints. Its robust motion controllability is validated
by drastic increases in the bounding box alignment metric. Human evaluation
also shows that users favor Boximator generation results over the base model.
Boximator enhances video synthesis by adding fine-grained motion control through box constraints, achieving superior video quality and alignment compared to base models.
Boximator hard box soft box video diffusion models control module self-tracking FVD scores bounding box alignment