TrackGo: A Flexible and Efficient Method for Controllable Video
  Generation
Recent years have seen substantial progress in diffusion-based controllable
video generation. However, achieving precise control in complex scenarios,
including fine-grained object parts, sophisticated motion trajectories, and
coherent background movement, remains a challenge. In this paper, we introduce
TrackGo, a novel approach that leverages free-form masks and arrows for
conditional video generation. This method offers users with a flexible and
precise mechanism for manipulating video content. We also propose the
TrackAdapter for control implementation, an efficient and lightweight adapter
designed to be seamlessly integrated into the temporal self-attention layers of
a pretrained video generation model. This design leverages our observation that
the attention map of these layers can accurately activate regions corresponding
to motion in videos. Our experimental results demonstrate that our new
approach, enhanced by the TrackAdapter, achieves state-of-the-art performance
on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo
can be found at: https://zhtjtcz.github.io/TrackGo-Page/
66c6a4afc33122b03a2c1c27
Haitao Zhou
claimed_verified
2024-12-11T09:54:21.578Z
6474352dd9e9598973cd8831
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/B_MnqQSvlkodzgf9krjF7.jpeg
Haitao Zhou
user
Marvolo
66c6a4afc33122b03a2c1c28
Chuang Wang
66c6a4afc33122b03a2c1c29
Rui Nie
66c6a4afc33122b03a2c1c2a
Jinxiao Lin
66c6a4afc33122b03a2c1c2b
Dongdong Yu
66c6a4afc33122b03a2c1c2c
Qian Yu
66c6a4afc33122b03a2c1c2d
Changhu Wang
TrackGo: A Flexible and Efficient Method for Controllable Video
  Generation
Recent years have seen substantial progress in diffusion-based controllable
video generation. However, achieving precise control in complex scenarios,
including fine-grained object parts, sophisticated motion trajectories, and
coherent background movement, remains a challenge. In this paper, we introduce
TrackGo, a novel approach that leverages free-form masks and arrows for
conditional video generation. This method offers users with a flexible and
precise mechanism for manipulating video content. We also propose the
TrackAdapter for control implementation, an efficient and lightweight adapter
designed to be seamlessly integrated into the temporal self-attention layers of
a pretrained video generation model. This design leverages our observation that
the attention map of these layers can accurately activate regions corresponding
to motion in videos. Our experimental results demonstrate that our new
approach, enhanced by the TrackAdapter, achieves state-of-the-art performance
on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo
can be found at: https://zhtjtcz.github.io/TrackGo-Page/
TrackGo, using free-form masks and arrows along with TrackAdapter, achieves top performance in controlled video generation by enhancing pretrained temporal self-attention layers.
diffusion-based video generation fine-grained object parts sophisticated motion trajectories coherent background movement conditional video generation TrackGo TrackAdapter temporal self-attention layers FVD FID ObjMC scores