CoTracker: It is Better to Track Together
Methods for video motion prediction either estimate jointly the instantaneous
motion of all points in a given video frame using optical flow or independently
track the motion of individual points throughout the video. The latter is true
even for powerful deep-learning methods that can track points through
occlusions. Tracking points individually ignores the strong correlation that
can exist between the points, for instance, because they belong to the same
physical object, potentially harming performance. In this paper, we thus
propose CoTracker, an architecture that jointly tracks multiple points
throughout an entire video. This architecture combines several ideas from the
optical flow and tracking literature in a new, flexible and powerful design. It
is based on a transformer network that models the correlation of different
points in time via specialised attention layers. The transformer iteratively
updates an estimate of several trajectories. It can be applied in a
sliding-window manner to very long videos, for which we engineer an unrolled
training loop. It can track from one to several points jointly and supports
adding new points to track at any time. The result is a flexible and powerful
tracking algorithm that outperforms state-of-the-art methods in almost all
benchmarks.
64b5feb57d9d8f9970fa4961
Nikita Karaev
admin_assigned
2023-07-18T08:28:46.099Z
6393a73584c565d2c3416cb9
https://cdn-avatars.huggingface.co/v1/production/uploads/6393a73584c565d2c3416cb9/OGtX-i-_MLg5--qA054ti.jpeg
Nikita Karaev
user
nikkar
64b5feb57d9d8f9970fa4962
Ignacio Rocco
64b5feb57d9d8f9970fa4963
Benjamin Graham
64b5feb57d9d8f9970fa4964
Natalia Neverova
64b5feb57d9d8f9970fa4965
Andrea Vedaldi
64b5feb57d9d8f9970fa4966
Christian Rupprecht
CoTracker: It is Better to Track Together
Methods for video motion prediction either estimate jointly the instantaneous
motion of all points in a given video frame using optical flow or independently
track the motion of individual points throughout the video. The latter is true
even for powerful deep-learning methods that can track points through
occlusions. Tracking points individually ignores the strong correlation that
can exist between the points, for instance, because they belong to the same
physical object, potentially harming performance. In this paper, we thus
propose CoTracker, an architecture that jointly tracks multiple points
throughout an entire video. This architecture combines several ideas from the
optical flow and tracking literature in a new, flexible and powerful design. It
is based on a transformer network that models the correlation of different
points in time via specialised attention layers. The transformer iteratively
updates an estimate of several trajectories. It can be applied in a
sliding-window manner to very long videos, for which we engineer an unrolled
training loop. It can track from one to several points jointly and supports
adding new points to track at any time. The result is a flexible and powerful
tracking algorithm that outperforms state-of-the-art methods in almost all
benchmarks.
CoTracker, a deep learning architecture combining transformer networks and specialized attention layers, jointly tracks multiple video points across frames, offering superior performance compared to existing methods.
optical flow tracking CoTracker transformer network specialized attention layers trajectories sliding-window unrolled training loop