Advancing Reference-free Evaluation of Video Captions with Factual
  Analysis
Video captions offer concise snapshots of actors, objects, and actions within
a video, serving as valuable assets for applications such as question answering
and event localization. However, acquiring human annotations for video captions
is costly or even impractical, especially when dealing with diverse video
domains. Existing models trained on supervised datasets face challenges in
evaluating performance across different domains due to the reliance on
reference-based evaluation protocols, which necessitate ground truth captions.
This assumption is unrealistic for evaluating videos in the wild. To address
these limitations, we propose a reference-free evaluation framework that does
not require ground truth captions, focusing on factual grounding to ensure
accurate assessment of caption quality. We introduce VC-Inspector, a novel
caption quality evaluator that is both reference-free and factually grounded.
Utilizing large language models, we generate pseudo captions of varying quality
based on supervised data, which are subsequently used to train a multimodal
model (i.e., Qwen2.5-VL) as the evaluator. Our approach demonstrates superior
alignment with human judgments on the VATEX-Eval dataset, outperforming
existing methods. The performance also generalizes to image caption datasets,
Flickr8K-Expert and Flickr8K-CF, when viewing images as 1-frame videos.
Overall, VC-Inspector offers a scalable and generalizable solution for
evaluating the factual accuracy of video captions, paving the way for more
effective and objective assessment methodologies in diverse video domains.
68db698dd2bf1f4b15ec7655
Shubhashis Roy Dipta
claimed_verified
2025-09-30T06:00:07.361Z
638940232a897944ea5d2a41
https://cdn-avatars.huggingface.co/v1/production/uploads/638940232a897944ea5d2a41/JSsRSOhkKhuhRKr1CpFB9.jpeg
Shubhashis Roy Dipta
user
dipta007
68db698dd2bf1f4b15ec7656
Tz-Ying Wu
68db698dd2bf1f4b15ec7657
Subarna Tripathi
Advancing Reference-free Evaluation of Video Captions with Factual
  Analysis
Video captions offer concise snapshots of actors, objects, and actions within
a video, serving as valuable assets for applications such as question answering
and event localization. However, acquiring human annotations for video captions
is costly or even impractical, especially when dealing with diverse video
domains. Existing models trained on supervised datasets face challenges in
evaluating performance across different domains due to the reliance on
reference-based evaluation protocols, which necessitate ground truth captions.
This assumption is unrealistic for evaluating videos in the wild. To address
these limitations, we propose a reference-free evaluation framework that does
not require ground truth captions, focusing on factual grounding to ensure
accurate assessment of caption quality. We introduce VC-Inspector, a novel
caption quality evaluator that is both reference-free and factually grounded.
Utilizing large language models, we generate pseudo captions of varying quality
based on supervised data, which are subsequently used to train a multimodal
model (i.e., Qwen2.5-VL) as the evaluator. Our approach demonstrates superior
alignment with human judgments on the VATEX-Eval dataset, outperforming
existing methods. The performance also generalizes to image caption datasets,
Flickr8K-Expert and Flickr8K-CF, when viewing images as 1-frame videos.
Overall, VC-Inspector offers a scalable and generalizable solution for
evaluating the factual accuracy of video captions, paving the way for more
effective and objective assessment methodologies in diverse video domains.
VC-Inspector, a reference-free and factually grounded caption quality evaluator, uses large language models to generate pseudo captions and train a multimodal model, demonstrating superior performance in evaluating video captions across diverse domains.
reference-free evaluation factually grounded caption quality evaluator large language models multimodal model VATEX-Eval dataset Flickr8K-Expert Flickr8K-CF