Q-Eval-100K: Evaluating Visual Quality and Alignment Level for
  Text-to-Vision Content
Evaluating text-to-vision content hinges on two crucial aspects: visual
quality and alignment. While significant progress has been made in developing
objective models to assess these dimensions, the performance of such models
heavily relies on the scale and quality of human annotations. According to
Scaling Law, increasing the number of human-labeled instances follows a
predictable pattern that enhances the performance of evaluation models.
Therefore, we introduce a comprehensive dataset designed to Evaluate Visual
quality and Alignment Level for text-to-vision content (Q-EVAL-100K), featuring
the largest collection of human-labeled Mean Opinion Scores (MOS) for the
mentioned two aspects. The Q-EVAL-100K dataset encompasses both text-to-image
and text-to-video models, with 960K human annotations specifically focused on
visual quality and alignment for 100K instances (60K images and 40K videos).
Leveraging this dataset with context prompt, we propose Q-Eval-Score, a unified
model capable of evaluating both visual quality and alignment with special
improvements for handling long-text prompt alignment. Experimental results
indicate that the proposed Q-Eval-Score achieves superior performance on both
visual quality and alignment, with strong generalization capabilities across
other benchmarks. These findings highlight the significant value of the
Q-EVAL-100K dataset. Data and codes will be available at
https://github.com/zzc-1998/Q-Eval.
67c830966f68759bf38567a5
Zicheng Zhang
claimed_verified
2025-04-08T06:58:03.826Z
6526cc6bab4f5d98382f5603
https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
Zicheng Zhang
user
zhangzicheng
67c830966f68759bf38567a6
Tengchuan Kou
67c830966f68759bf38567a7
Shushi Wang
admin_assigned
2025-03-05T11:36:18.075Z
64687fb52a180a1378359c46
/avatars/746ea4d52801227fd44b9a9f380738a2.svg
shushi wang 
user
wcain
67c830966f68759bf38567a8
Chunyi Li
67c830966f68759bf38567a9
Wei Sun
67c830966f68759bf38567aa
Wei Wang
67c830966f68759bf38567ab
Xiaoyu Li
67c830966f68759bf38567ac
Zongyu Wang
admin_assigned
2025-03-05T11:36:37.427Z
6459f4f9667a889f3d35f51c
/avatars/a4a407b8f0d6e3616c8e7ae7b4c78a7d.svg
zongyu wang
user
sl2782087
67c830966f68759bf38567ad
Xuezhi Cao
admin_assigned
2025-03-05T11:36:30.233Z
647095cdcf0a68ad56aa7181
/avatars/65e87b38b90c57b6db82e51f48a15fcf.svg
xuezhi cao
user
a9108
67c830966f68759bf38567ae
Xiongkuo Min
67c830966f68759bf38567af
Xiaohong Liu
67c830966f68759bf38567b0
Guangtao Zhai
admin_assigned
2025-03-05T11:37:03.268Z
65535b125413c1a54e6fb243
/avatars/03bcf1d58865f5406aff49a415e78bdc.svg
Guangtao Zhai
user
GTZhai
Q-Eval-100K: Evaluating Visual Quality and Alignment Level for
  Text-to-Vision Content
Evaluating text-to-vision content hinges on two crucial aspects: visual
quality and alignment. While significant progress has been made in developing
objective models to assess these dimensions, the performance of such models
heavily relies on the scale and quality of human annotations. According to
Scaling Law, increasing the number of human-labeled instances follows a
predictable pattern that enhances the performance of evaluation models.
Therefore, we introduce a comprehensive dataset designed to Evaluate Visual
quality and Alignment Level for text-to-vision content (Q-EVAL-100K), featuring
the largest collection of human-labeled Mean Opinion Scores (MOS) for the
mentioned two aspects. The Q-EVAL-100K dataset encompasses both text-to-image
and text-to-video models, with 960K human annotations specifically focused on
visual quality and alignment for 100K instances (60K images and 40K videos).
Leveraging this dataset with context prompt, we propose Q-Eval-Score, a unified
model capable of evaluating both visual quality and alignment with special
improvements for handling long-text prompt alignment. Experimental results
indicate that the proposed Q-Eval-Score achieves superior performance on both
visual quality and alignment, with strong generalization capabilities across
other benchmarks. These findings highlight the significant value of the
Q-EVAL-100K dataset. Data and codes will be available at
https://github.com/zzc-1998/Q-Eval.
A large-scale dataset with human-labeled MOS for visual quality and alignment enhances a unified text-to-vision evaluation model's performance.
text-to-image text-to-video Mean Opinion Scores (MOS) unified model long-text prompt alignment evaluation model visual quality alignment generalization capabilities benchmarks