VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC
  Videos
MLLMs have been widely studied for video question answering recently.
However, most existing assessments focus on natural videos, overlooking
synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in
video generation rely on MLLMs to evaluate the quality of generated videos, but
the capabilities of MLLMs on interpreting AIGC videos remain largely
underexplored. To address this, we propose a new benchmark, VF-Eval, which
introduces four tasks-coherence validation, error awareness, error type
detection, and reasoning evaluation-to comprehensively evaluate the abilities
of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that
even the best-performing model, GPT-4.1, struggles to achieve consistently good
performance across all tasks. This highlights the challenging nature of our
benchmark. Additionally, to investigate the practical applications of VF-Eval
in improving video generation, we conduct an experiment, RePrompt,
demonstrating that aligning MLLMs more closely with human feedback can benefit
video generation.
68390e95b85141ce6c11b50f
Tingyu Song
claimed_verified
2025-05-30T06:54:12.429Z
64dc29d9b5d625e0e9a6ecb9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
Tingyu Song
user
songtingyu
68390e95b85141ce6c11b510
Tongyan Hu
claimed_verified
2025-05-30T06:54:15.154Z
66e83ec5deb449d8d856e78d
/avatars/c5e56be65fcacb3192ce10ba6d8f48e2.svg
Tongyan Hu
user
entropyhu
68390e95b85141ce6c11b511
Guo Gan
68390e95b85141ce6c11b512
Yilun Zhao
claimed_verified
2025-06-02T07:47:26.006Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC
  Videos
MLLMs have been widely studied for video question answering recently.
However, most existing assessments focus on natural videos, overlooking
synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in
video generation rely on MLLMs to evaluate the quality of generated videos, but
the capabilities of MLLMs on interpreting AIGC videos remain largely
underexplored. To address this, we propose a new benchmark, VF-Eval, which
introduces four tasks-coherence validation, error awareness, error type
detection, and reasoning evaluation-to comprehensively evaluate the abilities
of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that
even the best-performing model, GPT-4.1, struggles to achieve consistently good
performance across all tasks. This highlights the challenging nature of our
benchmark. Additionally, to investigate the practical applications of VF-Eval
in improving video generation, we conduct an experiment, RePrompt,
demonstrating that aligning MLLMs more closely with human feedback can benefit
video generation.
A new benchmark, VF-Eval, evaluates the capabilities of MLLMs in interpreting AI-generated content videos across four tasks, highlighting challenges and demonstrating benefits in video generation through human feedback alignment.
MLLMs video question answering synthetic videos AI-generated content (AIGC) VF-Eval coherence validation error awareness error type detection reasoning evaluation GPT-4.1 RePrompt