No Time to Waste: Squeeze Time into Channel for Mobile Video
  Understanding
Current architectures for video understanding mainly build upon 3D
convolutional blocks or 2D convolutions with additional operations for temporal
modeling. However, these methods all regard the temporal axis as a separate
dimension of the video sequence, which requires large computation and memory
budgets and thus limits their usage on mobile devices. In this paper, we
propose to squeeze the time axis of a video sequence into the channel dimension
and present a lightweight video recognition network, term as
SqueezeTime, for mobile video understanding. To enhance the temporal
modeling capability of the proposed network, we design a Channel-Time Learning
(CTL) Block to capture temporal dynamics of the sequence. This module has two
complementary branches, in which one branch is for temporal importance learning
and another branch with temporal position restoring capability is to enhance
inter-temporal object modeling ability. The proposed SqueezeTime is much
lightweight and fast with high accuracies for mobile video understanding.
Extensive experiments on various video recognition and action detection
benchmarks, i.e., Kinetics400, Kinetics600, HMDB51, AVA2.1 and THUMOS14,
demonstrate the superiority of our model. For example, our SqueezeTime achieves
+1.2% accuracy and +80% GPU throughput gain on Kinetics400 than prior
methods. Codes are publicly available at
https://github.com/xinghaochen/SqueezeTime and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SqueezeTime.
6644fc9e8264e57bf9445843
Yingjie Zhai
6644fc9e8264e57bf9445844
Wenshuo Li
admin_assigned
2024-05-15T20:11:12.734Z
65f9312ac1d406d7161d6141
/avatars/b249df8e7e9fb1133365db2ce5bc3f03.svg
LiWenshuo
user
meteorh
6644fc9e8264e57bf9445845
Yehui Tang
admin_assigned
2024-05-15T20:11:20.999Z
64d5deb154bb9eb704f83122
/avatars/86ce09bcca903319051e2307581a43f4.svg
Yehui Tang
user
tangyehui
6644fc9e8264e57bf9445846
Xinghao Chen
claimed_verified
2024-05-16T10:43:02.545Z
65878c774f9d2b955e025019
/avatars/c039c29e131a8cba96b38048da94971b.svg
Xinghao Chen
user
xinghaochen
6644fc9e8264e57bf9445847
Yunhe Wang
admin_assigned
2024-05-15T20:12:06.407Z
658bdf7b925aadd43304f05c
/avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
Yunhe Wang
user
MightyCrane
No Time to Waste: Squeeze Time into Channel for Mobile Video
  Understanding
Current architectures for video understanding mainly build upon 3D
convolutional blocks or 2D convolutions with additional operations for temporal
modeling. However, these methods all regard the temporal axis as a separate
dimension of the video sequence, which requires large computation and memory
budgets and thus limits their usage on mobile devices. In this paper, we
propose to squeeze the time axis of a video sequence into the channel dimension
and present a lightweight video recognition network, term as
SqueezeTime, for mobile video understanding. To enhance the temporal
modeling capability of the proposed network, we design a Channel-Time Learning
(CTL) Block to capture temporal dynamics of the sequence. This module has two
complementary branches, in which one branch is for temporal importance learning
and another branch with temporal position restoring capability is to enhance
inter-temporal object modeling ability. The proposed SqueezeTime is much
lightweight and fast with high accuracies for mobile video understanding.
Extensive experiments on various video recognition and action detection
benchmarks, i.e., Kinetics400, Kinetics600, HMDB51, AVA2.1 and THUMOS14,
demonstrate the superiority of our model. For example, our SqueezeTime achieves
+1.2% accuracy and +80% GPU throughput gain on Kinetics400 than prior
methods. Codes are publicly available at
https://github.com/xinghaochen/SqueezeTime and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SqueezeTime.
The proposed SqueezeTime network enhances video understanding on mobile devices by integrating the temporal axis into the channel dimension, improving accuracy and GPU throughput.
3D convolutions 2D convolutions temporal modeling SqueezeTime Channel-Time Learning (CTL) Block temporal importance learning inter-temporal object modeling Kinetics400 Kinetics600 HMDB51 AVA2.1 THUMOS14