Multiscale Video Pretraining for Long-Term Activity Forecasting
Long-term activity forecasting is an especially challenging research problem
because it requires understanding the temporal relationships between observed
actions, as well as the variability and complexity of human activities. Despite
relying on strong supervision via expensive human annotations, state-of-the-art
forecasting approaches often generalize poorly to unseen data. To alleviate
this issue, we propose Multiscale Video Pretraining (MVP), a novel
self-supervised pretraining approach that learns robust representations for
forecasting by learning to predict contextualized representations of future
video clips over multiple timescales. MVP is based on our observation that
actions in videos have a multiscale nature, where atomic actions typically
occur at a short timescale and more complex actions may span longer timescales.
We compare MVP to state-of-the-art self-supervised video learning approaches on
downstream long-term forecasting tasks including long-term action anticipation
and video summary prediction. Our comprehensive experiments across the Ego4D
and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs
state-of-the-art methods by significant margins. Notably, MVP obtains a
relative performance gain of over 20% accuracy in video summary forecasting
over existing methods.
64bf90e5a0e54710669129bc
Reuben Tan
64bf90e5a0e54710669129bd
Matthias De Lange
64bf90e5a0e54710669129be
Michael Iuzzolino
64bf90e5a0e54710669129bf
Bryan A. Plummer
64bf90e5a0e54710669129c0
Kate Saenko
admin_assigned
2023-07-25T09:19:06.958Z
6272e87b72307af646a5372c
/avatars/258f3977ae88d747892dcf83b2c56c2e.svg
Kate Saenko
user
KateSaenko
64bf90e5a0e54710669129c1
Karl Ridgeway
64bf90e5a0e54710669129c2
Lorenzo Torresani
Multiscale Video Pretraining for Long-Term Activity Forecasting
Long-term activity forecasting is an especially challenging research problem
because it requires understanding the temporal relationships between observed
actions, as well as the variability and complexity of human activities. Despite
relying on strong supervision via expensive human annotations, state-of-the-art
forecasting approaches often generalize poorly to unseen data. To alleviate
this issue, we propose Multiscale Video Pretraining (MVP), a novel
self-supervised pretraining approach that learns robust representations for
forecasting by learning to predict contextualized representations of future
video clips over multiple timescales. MVP is based on our observation that
actions in videos have a multiscale nature, where atomic actions typically
occur at a short timescale and more complex actions may span longer timescales.
We compare MVP to state-of-the-art self-supervised video learning approaches on
downstream long-term forecasting tasks including long-term action anticipation
and video summary prediction. Our comprehensive experiments across the Ego4D
and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs
state-of-the-art methods by significant margins. Notably, MVP obtains a
relative performance gain of over 20% accuracy in video summary forecasting
over existing methods.
Multiscale Video Pretraining (MVP) enhances long-term activity forecasting by learning robust representations across different timescales, improving accuracy on tasks like action anticipation and video summary prediction.
Multiscale Video Pretraining MVP self-supervised pretraining contextualized representations timescales Ego4D Epic-Kitchens-55/100 long-term forecasting action anticipation video summary prediction