CoS: Chain-of-Shot Prompting for Long Video Understanding
Multi-modal Large Language Models (MLLMs) struggle with long videos due to
the need for excessive visual tokens. These tokens exceed massively the context
length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to
select shots is an unsolved critical problem: sparse sampling risks missing key
details, while exhaustive sampling overwhelms the model with irrelevant
content, leading to video misunderstanding. To solve this problem, we propose
Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as
test-time visual prompt optimisation, choosing shots adaptive to video
understanding semantic task by optimising shots-task alignment. CoS has two key
parts: (1) a binary video summary mechanism that performs pseudo temporal
grounding, discovering a binary coding to identify task-relevant shots, and (2)
a video co-reasoning module that deploys the binary coding to pair (learning to
align) task-relevant positive shots with irrelevant negative shots. It embeds
the optimised shot selections into the original video, facilitating a focus on
relevant context to optimize long video understanding. Experiments across three
baselines and five datasets demonstrate the effectiveness and adaptability of
CoS. Code given in https://lwpyh.github.io/CoS.
67ac99089e12456bdb1d2e9d
Jian Hu
claimed_verified
2025-02-19T11:12:25.146Z
65e1b6e9501590df0173cbd3
/avatars/a73e2139700e23eff455734c99cef5ba.svg
Jian Hu
user
lwpyh
67ac99089e12456bdb1d2e9e
Zixu Cheng
67ac99089e12456bdb1d2e9f
Chenyang Si
67ac99089e12456bdb1d2ea0
Wei Li
67ac99089e12456bdb1d2ea1
Shaogang Gong
CoS: Chain-of-Shot Prompting for Long Video Understanding
Multi-modal Large Language Models (MLLMs) struggle with long videos due to
the need for excessive visual tokens. These tokens exceed massively the context
length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to
select shots is an unsolved critical problem: sparse sampling risks missing key
details, while exhaustive sampling overwhelms the model with irrelevant
content, leading to video misunderstanding. To solve this problem, we propose
Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as
test-time visual prompt optimisation, choosing shots adaptive to video
understanding semantic task by optimising shots-task alignment. CoS has two key
parts: (1) a binary video summary mechanism that performs pseudo temporal
grounding, discovering a binary coding to identify task-relevant shots, and (2)
a video co-reasoning module that deploys the binary coding to pair (learning to
align) task-relevant positive shots with irrelevant negative shots. It embeds
the optimised shot selections into the original video, facilitating a focus on
relevant context to optimize long video understanding. Experiments across three
baselines and five datasets demonstrate the effectiveness and adaptability of
CoS. Code given in https://lwpyh.github.io/CoS.
Chain-of-Shot prompting optimizes shot selection for long video understanding by aligning relevant shots with the video's semantic task, enhancing model performance without redundancy.
multi-modal large language models MLLMs long videos visual tokens context length sparse sampling exhaustive sampling shot selection test-time visual prompt optimisation shots-task alignment binary video summary pseudo temporal grounding video co-reasoning module task-relevant shots irrelevant shots long video understanding