MantisScore: Building Automatic Metrics to Simulate Fine-grained Human
  Feedback for Video Generation
The recent years have witnessed great advances in video generation. However,
the development of automatic video metrics is lagging significantly behind.
None of the existing metric is able to provide reliable scores over generated
videos. The main barrier is the lack of large-scale human-annotated dataset. In
this paper, we release VideoFeedback, the first large-scale dataset containing
human-provided multi-aspect score over 37.6K synthesized videos from 11
existing video generative models. We train MantisScore (initialized from
Mantis) based on VideoFeedback to enable automatic video quality assessment.
Experiments show that the Spearman correlation between MantisScore and humans
can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about
50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and
VBench show that MantisScore has consistently much higher correlation with
human judges than other metrics. Due to these results, we believe MantisScore
can serve as a great proxy for human raters to (1) rate different video models
to track progress (2) simulate fine-grained human feedback in Reinforcement
Learning with Human Feedback (RLHF) to improve current video generation models.
6678e7e5405bf950a0f06973
Xuan He
claimed_verified
2024-09-25T07:35:33.469Z
655c3953ce055ed40a7de0ba
/avatars/70aad195f6aff83279ef1b1f8859419d.svg
Xuan He
user
hexuan21
6678e7e5405bf950a0f06974
Dongfu Jiang
admin_assigned
2024-06-24T14:13:28.524Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
6678e7e5405bf950a0f06975
Ge Zhang
admin_assigned
2024-06-24T14:13:41.373Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6678e7e5405bf950a0f06976
Max Ku
claimed_verified
2024-06-25T07:34:02.711Z
631d760344503b7227837242
https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
Max Ku
user
vinesmsuic
6678e7e5405bf950a0f06977
Achint Soni
admin_assigned
2024-06-24T14:13:56.236Z
6584bd1a5b94130d87c5f701
/avatars/615e392f98e763144736c363f980a85a.svg
Achint Soni
user
Trickyjustice
6678e7e5405bf950a0f06978
Sherman Siu
admin_assigned
2024-06-24T14:14:04.893Z
630d73387dacb93b335b00e9
/avatars/2ded3550aedf77a58175a66319d01820.svg
Sherman Siu
user
shermansiu
6678e7e5405bf950a0f06979
Haonan Chen
6678e7e5405bf950a0f0697a
Abhranil Chandra
admin_assigned
2024-06-24T14:14:36.268Z
6154ee656181394cc00cb990
/avatars/25d3b6911c5991b1869f5d76ca2c4069.svg
Abhranil Chandra
user
abhranil14
6678e7e5405bf950a0f0697b
Ziyan Jiang
claimed_verified
2024-06-24T07:27:45.178Z
64778fb8168cb428e00f69b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
Ziyan Jiang
user
ziyjiang
6678e7e5405bf950a0f0697c
Aaran Arulraj
admin_assigned
2024-06-24T14:14:42.671Z
65fb7212fc9132a2dfb80061
/avatars/7955a4c96cba4e5b25481943f7b84be1.svg
Aaran Arulraj
user
AaranArulraj
6678e7e5405bf950a0f0697d
Kai Wang
6678e7e5405bf950a0f0697e
Quy Duc Do
admin_assigned
2024-06-24T14:14:51.954Z
65fa5fa1180d06d37c845d51
/avatars/a04eb59f0a3a3b5800ae443a8eaab785.svg
Quy Duc Do
user
qd2do
6678e7e5405bf950a0f0697f
Yuansheng Ni
admin_assigned
2024-06-24T14:14:57.984Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
6678e7e5405bf950a0f06980
Bohan Lyu
claimed_verified
2024-11-04T09:36:33.224Z
650267e7e751d03da933a24a
/avatars/f047a047d1de304cd97027463541bdf3.svg
Bohan22
user
Bohan22
6678e7e5405bf950a0f06981
Yaswanth Narsupalli
claimed_verified
2025-01-10T08:22:11.239Z
64f6dc785f2dee8a6b9fcada
/avatars/7e9930c141eebdc9fcaddf2fd1d3a53c.svg
Narsupalli Yaswanth
user
yaswanth-iitkgp
6678e7e5405bf950a0f06982
Rongqi Fan
6678e7e5405bf950a0f06983
Zhiheng Lyu
6678e7e5405bf950a0f06984
Yuchen Lin
admin_assigned
2024-06-24T14:15:33.764Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
6678e7e5405bf950a0f06985
Wenhu Chen
admin_assigned
2024-06-24T14:15:39.383Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
MantisScore: Building Automatic Metrics to Simulate Fine-grained Human
  Feedback for Video Generation
The recent years have witnessed great advances in video generation. However,
the development of automatic video metrics is lagging significantly behind.
None of the existing metric is able to provide reliable scores over generated
videos. The main barrier is the lack of large-scale human-annotated dataset. In
this paper, we release VideoFeedback, the first large-scale dataset containing
human-provided multi-aspect score over 37.6K synthesized videos from 11
existing video generative models. We train MantisScore (initialized from
Mantis) based on VideoFeedback to enable automatic video quality assessment.
Experiments show that the Spearman correlation between MantisScore and humans
can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about
50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and
VBench show that MantisScore has consistently much higher correlation with
human judges than other metrics. Due to these results, we believe MantisScore
can serve as a great proxy for human raters to (1) rate different video models
to track progress (2) simulate fine-grained human feedback in Reinforcement
Learning with Human Feedback (RLHF) to improve current video generation models.
MantisScore, trained on the VideoFeedback dataset, provides highly accurate automatic video quality assessment, outperforming existing metrics and serving as a reliable human-rater proxy.
VideoFeedback MantisScore large-scale dataset human-provided scores video generative models Spearman correlation EvalCrafter GenAI-Bench VBench Reinforcement Learning with Human Feedback (RLHF)