EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the
  Backbone
Video-language pre-training (VLP) has become increasingly important due to
its ability to generalize to various vision and language tasks. However,
existing egocentric VLP frameworks utilize separate video and language encoders
and learn task-specific cross-modal information only during fine-tuning,
limiting the development of a unified system. In this work, we introduce the
second generation of egocentric video-language pre-training (EgoVLPv2), a
significant improvement from the previous generation, by incorporating
cross-modal fusion directly into the video and language backbones. EgoVLPv2
learns strong video-text representation during pre-training and reuses the
cross-modal attention modules to support different downstream tasks in a
flexible and efficient manner, reducing fine-tuning costs. Moreover, our
proposed fusion in the backbone strategy is more lightweight and
compute-efficient than stacking additional fusion-specific layers. Extensive
experiments on a wide range of VL tasks demonstrate the effectiveness of
EgoVLPv2 by achieving consistent state-of-the-art performance over strong
baselines across all downstream. Our project page can be found at
https://shramanpramanick.github.io/EgoVLPv2/.
64ae1a65692be9d12f8ca702
Shraman Pramanick
claimed_verified
2023-07-13T08:32:31.322Z
64aec10a9135f79b09117476
/avatars/83db978a906782b6df71d178b7a0df89.svg
Shraman Pramanick
user
Shraman
64ae1a65692be9d12f8ca703
Yale Song
64ae1a65692be9d12f8ca704
Sayan Nag
claimed_verified
2023-07-13T08:32:37.150Z
61de7e35c201b96c5db3bec6
/avatars/d69c9fcbeff60f6f1c442b2a1226307a.svg
Sayan Nag
user
nagsayan
64ae1a65692be9d12f8ca705
Kevin Qinghong Lin
claimed_verified
2024-01-04T09:33:58.364Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
64ae1a65692be9d12f8ca706
Hardik Shah
64ae1a65692be9d12f8ca707
Mike Zheng Shou
64ae1a65692be9d12f8ca708
Rama Chellappa
64ae1a65692be9d12f8ca709
Pengchuan Zhang
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the
  Backbone
Video-language pre-training (VLP) has become increasingly important due to
its ability to generalize to various vision and language tasks. However,
existing egocentric VLP frameworks utilize separate video and language encoders
and learn task-specific cross-modal information only during fine-tuning,
limiting the development of a unified system. In this work, we introduce the
second generation of egocentric video-language pre-training (EgoVLPv2), a
significant improvement from the previous generation, by incorporating
cross-modal fusion directly into the video and language backbones. EgoVLPv2
learns strong video-text representation during pre-training and reuses the
cross-modal attention modules to support different downstream tasks in a
flexible and efficient manner, reducing fine-tuning costs. Moreover, our
proposed fusion in the backbone strategy is more lightweight and
compute-efficient than stacking additional fusion-specific layers. Extensive
experiments on a wide range of VL tasks demonstrate the effectiveness of
EgoVLPv2 by achieving consistent state-of-the-art performance over strong
baselines across all downstream. Our project page can be found at
https://shramanpramanick.github.io/EgoVLPv2/.
EgoVLPv2 improves video-language pre-training by integrating cross-modal fusion directly into the backbones, enhancing representation and reducing fine-tuning costs across various tasks.
egocentric VLP cross-modal fusion video-text representation cross-modal attention modules fine-tuning costs VL tasks