Tracktention: Leveraging Point Tracking to Attend Videos Faster and
  Better
Temporal consistency is critical in video prediction to ensure that outputs
are coherent and free of artifacts. Traditional methods, such as temporal
attention and 3D convolution, may struggle with significant object motion and
may not capture long-range temporal dependencies in dynamic scenes. To address
this gap, we propose the Tracktention Layer, a novel architectural component
that explicitly integrates motion information using point tracks, i.e.,
sequences of corresponding points across frames. By incorporating these motion
cues, the Tracktention Layer enhances temporal alignment and effectively
handles complex object motions, maintaining consistent feature representations
over time. Our approach is computationally efficient and can be seamlessly
integrated into existing models, such as Vision Transformers, with minimal
modification. It can be used to upgrade image-only models to state-of-the-art
video ones, sometimes outperforming models natively designed for video
prediction. We demonstrate this on video depth prediction and video
colorization, where models augmented with the Tracktention Layer exhibit
significantly improved temporal consistency compared to baselines.
67e6ec0188bae9650ccefa02
Zihang Lai
admin_assigned
2025-03-31T08:21:29.995Z
64b6fc043240387159385543
/avatars/a63c688d176758ee826b3d02c11db8a9.svg
Zihang Lai
user
zlai
67e6ec0188bae9650ccefa03
Andrea Vedaldi
Tracktention: Leveraging Point Tracking to Attend Videos Faster and
  Better
Temporal consistency is critical in video prediction to ensure that outputs
are coherent and free of artifacts. Traditional methods, such as temporal
attention and 3D convolution, may struggle with significant object motion and
may not capture long-range temporal dependencies in dynamic scenes. To address
this gap, we propose the Tracktention Layer, a novel architectural component
that explicitly integrates motion information using point tracks, i.e.,
sequences of corresponding points across frames. By incorporating these motion
cues, the Tracktention Layer enhances temporal alignment and effectively
handles complex object motions, maintaining consistent feature representations
over time. Our approach is computationally efficient and can be seamlessly
integrated into existing models, such as Vision Transformers, with minimal
modification. It can be used to upgrade image-only models to state-of-the-art
video ones, sometimes outperforming models natively designed for video
prediction. We demonstrate this on video depth prediction and video
colorization, where models augmented with the Tracktention Layer exhibit
significantly improved temporal consistency compared to baselines.
The Tracktention Layer, which uses point tracks to integrate motion information, enhances temporal consistency and efficiency in video prediction tasks like depth estimation and colorization.
Temporal attention 3D convolution Tracktention Layer point tracks temporal alignment Vision Transformers video prediction video depth prediction video colorization