SIGMA: Sinkhorn-Guided Masked Video Modeling
Video-based pretraining offers immense potential for learning strong visual
representations on an unprecedented scale. Recently, masked video modeling
methods have shown promising scalability, yet fall short in capturing
higher-level semantics due to reconstructing predefined low-level targets such
as pixels. To tackle this, we present Sinkhorn-guided Masked Video Modelling
(SIGMA), a novel video pretraining method that jointly learns the video model
in addition to a target feature space using a projection network. However, this
simple modification means that the regular L2 reconstruction loss will lead to
trivial solutions as both networks are jointly optimized. As a solution, we
distribute features of space-time tubes evenly across a limited number of
learnable clusters. By posing this as an optimal transport problem, we enforce
high entropy in the generated features across the batch, infusing semantic and
temporal meaning into the feature space. The resulting cluster assignments are
used as targets for a symmetric prediction task where the video model predicts
cluster assignment of the projection network and vice versa. Experimental
results on ten datasets across three benchmarks validate the effectiveness of
SIGMA in learning more performant, temporally-aware, and robust video
representations improving upon state-of-the-art methods. Our project website
with code is available at: https://quva-lab.github.io/SIGMA.
66a0b2d9684c17d8793c8a7c
Mohammadreza Salehi
claimed_verified
2024-07-24T08:12:04.687Z
66990ec3aa500cd99d640a19
/avatars/3520cab2b946158b2cdf6ed84f9516e0.svg
Mohammadreza Salehi
user
SMSD75
66a0b2d9684c17d8793c8a7d
Michael Dorkenwald
claimed_verified
2024-07-24T08:01:01.006Z
6566436e3d526e3297c1b4a1
/avatars/e433ffccb0f9c2d91649f9988655fedd.svg
Michael Dorkenwald
user
mdorkenw
66a0b2d9684c17d8793c8a7e
Fida Mohammad Thoker
66a0b2d9684c17d8793c8a7f
Efstratios Gavves
66a0b2d9684c17d8793c8a80
Cees G. M. Snoek
66a0b2d9684c17d8793c8a81
Yuki M. Asano
claimed_verified
2024-10-15T09:11:47.423Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
SIGMA: Sinkhorn-Guided Masked Video Modeling
Video-based pretraining offers immense potential for learning strong visual
representations on an unprecedented scale. Recently, masked video modeling
methods have shown promising scalability, yet fall short in capturing
higher-level semantics due to reconstructing predefined low-level targets such
as pixels. To tackle this, we present Sinkhorn-guided Masked Video Modelling
(SIGMA), a novel video pretraining method that jointly learns the video model
in addition to a target feature space using a projection network. However, this
simple modification means that the regular L2 reconstruction loss will lead to
trivial solutions as both networks are jointly optimized. As a solution, we
distribute features of space-time tubes evenly across a limited number of
learnable clusters. By posing this as an optimal transport problem, we enforce
high entropy in the generated features across the batch, infusing semantic and
temporal meaning into the feature space. The resulting cluster assignments are
used as targets for a symmetric prediction task where the video model predicts
cluster assignment of the projection network and vice versa. Experimental
results on ten datasets across three benchmarks validate the effectiveness of
SIGMA in learning more performant, temporally-aware, and robust video
representations improving upon state-of-the-art methods. Our project website
with code is available at: https://quva-lab.github.io/SIGMA.
SIGMA, a novel video pretraining method, uses optimal transport to improve high-level semantic and temporal understanding in video representations compared to existing approaches.
masked video modeling Sinkhorn SIGMA projection network L2 reconstruction loss space-time tubes optimal transport high entropy feature space symmetric prediction task temporally-aware robust video representations