MovieChat: From Dense Token to Sparse Memory for Long Video
  Understanding
Recently, integrating video foundation models and large language models to
build a video understanding system overcoming the limitations of specific
pre-defined vision tasks. Yet, existing systems can only handle videos with
very few frames. For long videos, the computation complexity, memory cost, and
long-term temporal connection are the remaining challenges. Inspired by
Atkinson-Shiffrin memory model, we develop an memory mechanism including a
rapidly updated short-term memory and a compact thus sustained long-term
memory. We employ tokens in Transformers as the carriers of memory. MovieChat
achieves state-of-the-art performace in long video understanding.
64c86c213137cc529dddb414
Enxin Song
admin_assigned
2023-08-01T09:30:28.314Z
63a7b196e27a6dbd4861e275
https://cdn-avatars.huggingface.co/v1/production/uploads/63a7b196e27a6dbd4861e275/QGn4hdDPNuSRgZVKxzDpq.jpeg
EnxinSong
user
Enxin
64c86c213137cc529dddb415
Wenhao Chai
admin_assigned
2023-08-01T09:33:29.799Z
637c7503fe115289cfecbe6b
https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
Wenhao Chai
user
wchai
64c86c213137cc529dddb416
Guanhong Wang
admin_assigned
2023-08-01T09:34:01.719Z
64b621c25b4a27b5dd1a252d
/avatars/93d984b29177e083e1416747005a3520.svg
Guanhong Wang
user
Rejoicelf
64c86c213137cc529dddb417
Yucheng Zhang
64c86c213137cc529dddb418
Haoyang Zhou
64c86c213137cc529dddb419
Feiyang Wu
64c86c213137cc529dddb41a
Xun Guo
admin_assigned
2023-08-01T09:35:13.711Z
64ae7ddf407a5cae8579c171
/avatars/f78e8958db16f5f5603ece527951ac23.svg
Xun Guo
user
xunguohf
64c86c213137cc529dddb41b
Tian Ye
claimed_verified
2024-10-13T20:21:26.183Z
66015e8aa4d296af07de538e
/avatars/a1295c631cc2646282c545859975ce4c.svg
Owen
user
Owen777
64c86c213137cc529dddb41c
Yan Lu
64c86c213137cc529dddb41d
Jenq-Neng Hwang
64c86c213137cc529dddb41e
Gaoang Wang
claimed_verified
2023-08-01T13:08:52.689Z
64c9033c5381684d3eaac7f1
/avatars/07d36ca193826044b0df04e3602b9ef8.svg
Gaoang Wang
user
GaoangWang
MovieChat: From Dense Token to Sparse Memory for Long Video
  Understanding
Recently, integrating video foundation models and large language models to
build a video understanding system overcoming the limitations of specific
pre-defined vision tasks. Yet, existing systems can only handle videos with
very few frames. For long videos, the computation complexity, memory cost, and
long-term temporal connection are the remaining challenges. Inspired by
Atkinson-Shiffrin memory model, we develop an memory mechanism including a
rapidly updated short-term memory and a compact thus sustained long-term
memory. We employ tokens in Transformers as the carriers of memory. MovieChat
achieves state-of-the-art performace in long video understanding.
A video understanding system using a dual-memory mechanism based on the Atkinson-Shiffrin model, leveraging Transformers, achieves leading performance on long videos.
video foundation models large language models long videos computation complexity memory cost long-term temporal connection Atkinson-Shiffrin memory model short-term memory long-term memory Transformers MovieChat