Fostering Video Reasoning via Next-Event Prediction
Next-token prediction serves as the foundational learning task enabling
reasoning in LLMs. But what should the learning task be when aiming to equip
MLLMs with temporal reasoning capabilities over video inputs? Existing tasks
such as video question answering often rely on annotations from humans or much
stronger MLLMs, while video captioning tends to entangle temporal reasoning
with spatial information. To address this gap, we propose next-event prediction
(NEP), a learning task that harnesses future video segments as a rich,
self-supervised signal to foster temporal reasoning. We segment each video into
past and future frames: the MLLM takes the past frames as input and predicts a
summary of events derived from the future frames, thereby encouraging the model
to reason temporally in order to complete the task. To support this task, we
curate V1-33K, a dataset comprising 33,000 automatically extracted video
segments spanning diverse real-world scenarios. We further explore a range of
video instruction-tuning strategies to study their effects on temporal
reasoning. To evaluate progress, we introduce FutureBench to assess coherence
in predicting unseen future events. Experiments validate that NEP offers a
scalable and effective training paradigm for fostering temporal reasoning in
MLLMs.
68380912e9c1608de91e23f3
Haonan Wang
68380912e9c1608de91e23f4
Hongfu Liu
68380912e9c1608de91e23f5
Xiangyan Liu
68380912e9c1608de91e23f6
Chao Du
68380912e9c1608de91e23f7
Kenji Kawaguchi
68380912e9c1608de91e23f8
Ye Wang
68380912e9c1608de91e23f9
Tianyu Pang
Fostering Video Reasoning via Next-Event Prediction
Next-token prediction serves as the foundational learning task enabling
reasoning in LLMs. But what should the learning task be when aiming to equip
MLLMs with temporal reasoning capabilities over video inputs? Existing tasks
such as video question answering often rely on annotations from humans or much
stronger MLLMs, while video captioning tends to entangle temporal reasoning
with spatial information. To address this gap, we propose next-event prediction
(NEP), a learning task that harnesses future video segments as a rich,
self-supervised signal to foster temporal reasoning. We segment each video into
past and future frames: the MLLM takes the past frames as input and predicts a
summary of events derived from the future frames, thereby encouraging the model
to reason temporally in order to complete the task. To support this task, we
curate V1-33K, a dataset comprising 33,000 automatically extracted video
segments spanning diverse real-world scenarios. We further explore a range of
video instruction-tuning strategies to study their effects on temporal
reasoning. To evaluate progress, we introduce FutureBench to assess coherence
in predicting unseen future events. Experiments validate that NEP offers a
scalable and effective training paradigm for fostering temporal reasoning in
MLLMs.
Next-event prediction (NEP) is proposed as a learning task to enable MLLMs to reason temporally over video inputs, using future video segments as a self-supervised signal.
next-token prediction next-event prediction (NEP) LLMs MLLMs video question answering video captioning temporal reasoning future video segments past frames video segments V1-33K video instruction-tuning strategies FutureBench