IF-VidCap: Can Video Caption Models Follow Instructions?
Although Multimodal Large Language Models (MLLMs) have demonstrated
proficiency in video captioning, practical applications require captions that
follow specific user instructions rather than generating exhaustive,
unconstrained descriptions. Current benchmarks, however, primarily assess
descriptive comprehensiveness while largely overlooking instruction-following
capabilities. To address this gap, we introduce IF-VidCap, a new benchmark for
evaluating controllable video captioning, which contains 1,400 high-quality
samples. Distinct from existing video captioning or general
instruction-following benchmarks, IF-VidCap incorporates a systematic framework
that assesses captions on two dimensions: format correctness and content
correctness. Our comprehensive evaluation of over 20 prominent models reveals a
nuanced landscape: despite the continued dominance of proprietary models, the
performance gap is closing, with top-tier open-source solutions now achieving
near-parity. Furthermore, we find that models specialized for dense captioning
underperform general-purpose MLLMs on complex instructions, indicating that
future work should simultaneously advance both descriptive richness and
instruction-following fidelity.
68f8390e7669bcaeecce0c7f
Shihao Li
claimed_verified
2025-10-22T15:44:17.090Z
67f9d060395fb1a0d7e4ae21
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GjpOfOuazN7IxcXBpVqRm.png
Shihao Li
user
Leexeo
68f8390e7669bcaeecce0c80
Yuanxing Zhang
68f8390e7669bcaeecce0c81
Jiangtao Wu
68f8390e7669bcaeecce0c82
Zhide Lei
68f8390e7669bcaeecce0c83
Yiwen He
68f8390e7669bcaeecce0c84
Runzhe Wen
68f8390e7669bcaeecce0c85
Chenxi Liao
68f8390e7669bcaeecce0c86
Chengkang Jiang
68f8390e7669bcaeecce0c87
An Ping
68f8390e7669bcaeecce0c88
Shuo Gao
68f8390e7669bcaeecce0c89
Suhan Wang
68f8390e7669bcaeecce0c8a
Zhaozhou Bian
68f8390e7669bcaeecce0c8b
Zijun Zhou
68f8390e7669bcaeecce0c8c
Jingyi Xie
68f8390e7669bcaeecce0c8d
Jiayi Zhou
68f8390e7669bcaeecce0c8e
Jing Wang
68f8390e7669bcaeecce0c8f
Yifan Yao
68f8390e7669bcaeecce0c90
Weihao Xie
68f8390e7669bcaeecce0c91
Yingshui Tan
68f8390e7669bcaeecce0c92
Yanghai Wang
68f8390e7669bcaeecce0c93
Qianqian Xie
68f8390e7669bcaeecce0c94
Zhaoxiang Zhang
68f8390e7669bcaeecce0c95
Jiaheng Liu
IF-VidCap: Can Video Caption Models Follow Instructions?
Although Multimodal Large Language Models (MLLMs) have demonstrated
proficiency in video captioning, practical applications require captions that
follow specific user instructions rather than generating exhaustive,
unconstrained descriptions. Current benchmarks, however, primarily assess
descriptive comprehensiveness while largely overlooking instruction-following
capabilities. To address this gap, we introduce IF-VidCap, a new benchmark for
evaluating controllable video captioning, which contains 1,400 high-quality
samples. Distinct from existing video captioning or general
instruction-following benchmarks, IF-VidCap incorporates a systematic framework
that assesses captions on two dimensions: format correctness and content
correctness. Our comprehensive evaluation of over 20 prominent models reveals a
nuanced landscape: despite the continued dominance of proprietary models, the
performance gap is closing, with top-tier open-source solutions now achieving
near-parity. Furthermore, we find that models specialized for dense captioning
underperform general-purpose MLLMs on complex instructions, indicating that
future work should simultaneously advance both descriptive richness and
instruction-following fidelity.
A new benchmark, IF-VidCap, evaluates video captioning models on instruction-following capabilities, revealing that top-tier open-source models are closing the performance gap with proprietary models.
Multimodal Large Language Models MLLMs video captioning instruction-following benchmark format correctness content correctness dense captioning general-purpose MLLMs