Video Action Differencing
How do two individuals differ when performing the same action? In this work,
we introduce Video Action Differencing (VidDiff), the novel task of identifying
subtle differences between videos of the same action, which has many
applications, such as coaching and skill learning. To enable development on
this new task, we first create VidDiffBench, a benchmark dataset containing 549
video pairs, with human annotations of 4,469 fine-grained action differences
and 2,075 localization timestamps indicating where these differences occur. Our
experiments demonstrate that VidDiffBench poses a significant challenge for
state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL.
By analyzing failure cases of LMMs on VidDiffBench, we highlight two key
challenges for this task: localizing relevant sub-actions over two videos and
fine-grained frame comparison. To overcome these, we propose the VidDiff
method, an agentic workflow that breaks the task into three stages: action
difference proposal, keyframe localization, and frame differencing, each stage
utilizing specialized foundation models. To encourage future research in this
new task, we release the benchmark at
https://huggingface.co/datasets/jmhb/VidDiffBench and code at
http://jmhb0.github.io/viddiff.
67d0e915d0038007e5a75178
James Burgess
claimed_verified
2025-03-12T08:41:30.547Z
650871aeb44445e9b3625c7b
https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
James Burgess
user
jmhb
67d0e915d0038007e5a75179
Xiaohan Wang
claimed_verified
2025-03-12T08:40:56.564Z
65703fab7f50602340d23704
/avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
Xiaohan Wang
user
nicholswang
67d0e915d0038007e5a7517a
Yuhui Zhang
claimed_verified
2025-04-29T07:59:31.208Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
user
yuhuizhang
67d0e915d0038007e5a7517b
Anita Rau
67d0e915d0038007e5a7517c
Alejandro Lozano
67d0e915d0038007e5a7517d
Lisa Dunlap
67d0e915d0038007e5a7517e
Trevor Darrell
67d0e915d0038007e5a7517f
Serena Yeung-Levy
Video Action Differencing
How do two individuals differ when performing the same action? In this work,
we introduce Video Action Differencing (VidDiff), the novel task of identifying
subtle differences between videos of the same action, which has many
applications, such as coaching and skill learning. To enable development on
this new task, we first create VidDiffBench, a benchmark dataset containing 549
video pairs, with human annotations of 4,469 fine-grained action differences
and 2,075 localization timestamps indicating where these differences occur. Our
experiments demonstrate that VidDiffBench poses a significant challenge for
state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL.
By analyzing failure cases of LMMs on VidDiffBench, we highlight two key
challenges for this task: localizing relevant sub-actions over two videos and
fine-grained frame comparison. To overcome these, we propose the VidDiff
method, an agentic workflow that breaks the task into three stages: action
difference proposal, keyframe localization, and frame differencing, each stage
utilizing specialized foundation models. To encourage future research in this
new task, we release the benchmark at
https://huggingface.co/datasets/jmhb/VidDiffBench and code at
http://jmhb0.github.io/viddiff.
VidDiff is a new task for identifying subtle differences between videos of the same action, addressed by a benchmark dataset and a three-stage method utilizing specialized foundation models.
Video Action Differencing VidDiff VidDiffBench benchmark dataset GPT-4o Qwen2-VL fine-grained action differences localization timestamps action difference proposal keyframe localization frame differencing foundation models