Point Prompting: Counterfactual Tracking with Video Diffusion Models
Trackers and video generators solve closely related problems: the former
analyze motion, while the latter synthesize it. We show that this connection
enables pretrained video diffusion models to perform zero-shot point tracking
by simply prompting them to visually mark points as they move over time. We
place a distinctively colored marker at the query point, then regenerate the
rest of the video from an intermediate noise level. This propagates the marker
across frames, tracing the point's trajectory. To ensure that the marker
remains visible in this counterfactual generation, despite such markers being
unlikely in natural videos, we use the unedited initial frame as a negative
prompt. Through experiments with multiple image-conditioned video diffusion
models, we find that these "emergent" tracks outperform those of prior
zero-shot methods and persist through occlusions, often obtaining performance
that is competitive with specialized self-supervised models.
68f0973e36f8b025381e1a6d
Ayush Shrivastava
68f0973e36f8b025381e1a6e
Sanyam Mehta
68f0973e36f8b025381e1a6f
Daniel Geng
68f0973e36f8b025381e1a70
Andrew Owens
Point Prompting: Counterfactual Tracking with Video Diffusion Models
Trackers and video generators solve closely related problems: the former
analyze motion, while the latter synthesize it. We show that this connection
enables pretrained video diffusion models to perform zero-shot point tracking
by simply prompting them to visually mark points as they move over time. We
place a distinctively colored marker at the query point, then regenerate the
rest of the video from an intermediate noise level. This propagates the marker
across frames, tracing the point's trajectory. To ensure that the marker
remains visible in this counterfactual generation, despite such markers being
unlikely in natural videos, we use the unedited initial frame as a negative
prompt. Through experiments with multiple image-conditioned video diffusion
models, we find that these "emergent" tracks outperform those of prior
zero-shot methods and persist through occlusions, often obtaining performance
that is competitive with specialized self-supervised models.
Pretrained video diffusion models can perform zero-shot point tracking by visually marking points and regenerating video frames, outperforming prior methods and handling occlusions.
video diffusion models zero-shot point tracking image-conditioned video diffusion models emergent tracks self-supervised models