Self-Supervised Any-Point Tracking by Contrastive Random Walks
We present a simple, self-supervised approach to the Tracking Any Point (TAP)
problem. We train a global matching transformer to find cycle consistent tracks
through video via contrastive random walks, using the transformer's
attention-based global matching to define the transition matrices for a random
walk on a space-time graph. The ability to perform "all pairs" comparisons
between points allows the model to obtain high spatial precision and to obtain
a strong contrastive learning signal, while avoiding many of the complexities
of recent approaches (such as coarse-to-fine matching). To do this, we propose
a number of design decisions that allow global matching architectures to be
trained through self-supervision using cycle consistency. For example, we
identify that transformer-based methods are sensitive to shortcut solutions,
and propose a data augmentation scheme to address them. Our method achieves
strong performance on the TapVid benchmarks, outperforming previous
self-supervised tracking methods, such as DIFT, and is competitive with several
supervised methods.
66f4cf4c8fa8e9549199f8ae
Ayush Shrivastava
claimed_verified
2024-09-26T08:23:02.260Z
66f41c37d8de0f55059d3317
/avatars/a730ff35ea1da9a2da79aa56617e82d8.svg
Ayush Shrivastava
user
ayshrv
66f4cf4c8fa8e9549199f8af
Andrew Owens
Self-Supervised Any-Point Tracking by Contrastive Random Walks
We present a simple, self-supervised approach to the Tracking Any Point (TAP)
problem. We train a global matching transformer to find cycle consistent tracks
through video via contrastive random walks, using the transformer's
attention-based global matching to define the transition matrices for a random
walk on a space-time graph. The ability to perform "all pairs" comparisons
between points allows the model to obtain high spatial precision and to obtain
a strong contrastive learning signal, while avoiding many of the complexities
of recent approaches (such as coarse-to-fine matching). To do this, we propose
a number of design decisions that allow global matching architectures to be
trained through self-supervision using cycle consistency. For example, we
identify that transformer-based methods are sensitive to shortcut solutions,
and propose a data augmentation scheme to address them. Our method achieves
strong performance on the TapVid benchmarks, outperforming previous
self-supervised tracking methods, such as DIFT, and is competitive with several
supervised methods.
A self-supervised method using a global matching transformer and contrastive random walks achieves strong performance in point tracking, surpassing other self-supervised methods and being competitive with supervised ones.
global matching transformer cycle consistent tracks contrastive random walks space-time graph transformer attention-based global matching transition matrices all pairs comparisons spatial precision contrastive learning data augmentation shortcut solutions TapVid benchmarks DIFT