VBench++: Comprehensive and Versatile Benchmark Suite for Video
  Generative Models
Video generation has witnessed significant advancements, yet evaluating these
models remains a challenge. A comprehensive evaluation benchmark for video
generation is indispensable for two reasons: 1) Existing metrics do not fully
align with human perceptions; 2) An ideal evaluation system should provide
insights to inform future developments of video generation. To this end, we
present VBench, a comprehensive benchmark suite that dissects "video generation
quality" into specific, hierarchical, and disentangled dimensions, each with
tailored prompts and evaluation methods. VBench has several appealing
properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
video generation (e.g., subject identity inconsistency, motion smoothness,
temporal flickering, and spatial relationship, etc). The evaluation metrics
with fine-grained levels reveal individual models' strengths and weaknesses. 2)
Human Alignment: We also provide a dataset of human preference annotations to
validate our benchmarks' alignment with human perception, for each evaluation
dimension respectively. 3) Valuable Insights: We look into current models'
ability across various evaluation dimensions, and various content types. We
also investigate the gaps between video and image generation models. 4)
Versatile Benchmarking: VBench++ supports evaluating text-to-video and
image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
ratio to enable fair evaluations across different image-to-video generation
settings. Beyond assessing technical quality, VBench++ evaluates the
trustworthiness of video generative models, providing a more holistic view of
model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
continually add new video generation models to our leaderboard to drive forward
the field of video generation.
673ecb860ebca74a17b0726f
Ziqi Huang
admin_assigned
2024-11-22T13:24:09.580Z
60efe7fa0d920bc7805cada5
https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
Ziqi Huang
user
Ziqi
673ecb860ebca74a17b07270
Fan Zhang
claimed_verified
2024-12-17T08:04:37.850Z
61f24cbb88b9b5abbe184a85
https://cdn-avatars.huggingface.co/v1/production/uploads/61f24cbb88b9b5abbe184a85/OvcJRU51yI8pdO77NBHLb.jpeg
zhangfan
user
Fan-s
673ecb860ebca74a17b07271
Xiaojie Xu
673ecb860ebca74a17b07272
Yinan He
claimed_verified
2025-03-21T11:49:50.139Z
62aafa49f29ff279b51f0182
https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
yinanhe
user
ynhe
673ecb860ebca74a17b07273
Jiashuo Yu
claimed_verified
2025-06-13T07:43:38.376Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
673ecb860ebca74a17b07274
Ziyue Dong
673ecb860ebca74a17b07275
Qianli Ma
claimed_verified
2025-03-21T12:46:04.603Z
6448b2f53e7b3c11be684348
https://cdn-avatars.huggingface.co/v1/production/uploads/6448b2f53e7b3c11be684348/QvlUQG3pWf8ZyEVBV6F7w.jpeg
Qianli Ma
user
Mqleet
673ecb860ebca74a17b07276
Nattapol Chanpaisit
673ecb860ebca74a17b07277
Chenyang Si
claimed_verified
2025-06-10T09:31:45.397Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
673ecb860ebca74a17b07278
Yuming Jiang
673ecb860ebca74a17b07279
Yaohui Wang
673ecb860ebca74a17b0727a
Xinyuan Chen
673ecb860ebca74a17b0727b
Ying-Cong Chen
admin_assigned
2024-11-22T13:24:50.254Z
655cba1d87b67834000590e8
/avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
Yingcong Chen
user
yingcongchen
673ecb860ebca74a17b0727c
Limin Wang
673ecb860ebca74a17b0727d
Dahua Lin
admin_assigned
2024-11-22T13:24:24.368Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
673ecb860ebca74a17b0727e
Yu Qiao
673ecb860ebca74a17b0727f
Ziwei Liu
admin_assigned
2024-11-22T13:24:18.620Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
VBench++: Comprehensive and Versatile Benchmark Suite for Video
  Generative Models
Video generation has witnessed significant advancements, yet evaluating these
models remains a challenge. A comprehensive evaluation benchmark for video
generation is indispensable for two reasons: 1) Existing metrics do not fully
align with human perceptions; 2) An ideal evaluation system should provide
insights to inform future developments of video generation. To this end, we
present VBench, a comprehensive benchmark suite that dissects "video generation
quality" into specific, hierarchical, and disentangled dimensions, each with
tailored prompts and evaluation methods. VBench has several appealing
properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in
video generation (e.g., subject identity inconsistency, motion smoothness,
temporal flickering, and spatial relationship, etc). The evaluation metrics
with fine-grained levels reveal individual models' strengths and weaknesses. 2)
Human Alignment: We also provide a dataset of human preference annotations to
validate our benchmarks' alignment with human perception, for each evaluation
dimension respectively. 3) Valuable Insights: We look into current models'
ability across various evaluation dimensions, and various content types. We
also investigate the gaps between video and image generation models. 4)
Versatile Benchmarking: VBench++ supports evaluating text-to-video and
image-to-video. We introduce a high-quality Image Suite with an adaptive aspect
ratio to enable fair evaluations across different image-to-video generation
settings. Beyond assessing technical quality, VBench++ evaluates the
trustworthiness of video generative models, providing a more holistic view of
model performance. 5) Full Open-Sourcing: We fully open-source VBench++ and
continually add new video generation models to our leaderboard to drive forward
the field of video generation.
VBench++ is a comprehensive benchmark suite for video generation that evaluates models across specific, hierarchical dimensions using fine-grained metrics and human preference annotations, offering insights into model strengths, weaknesses, and gaps compared to image generation models.
video generation quality subject identity inconsistency motion smoothness temporal flickering spatial relationship human preference annotations text-to-video image-to-video high-quality Image Suite adaptive aspect ratio trustworthiness video generative models