Evaluation Agent: Efficient and Promptable Evaluation Framework for
  Visual Generative Models
Recent advancements in visual generative models have enabled high-quality
image and video generation, opening diverse applications. However, evaluating
these models often demands sampling hundreds or thousands of images or videos,
making the process computationally expensive, especially for diffusion-based
models with inherently slow sampling. Moreover, existing evaluation methods
rely on rigid pipelines that overlook specific user needs and provide numerical
results without clear explanations. In contrast, humans can quickly form
impressions of a model's capabilities by observing only a few samples. To mimic
this, we propose the Evaluation Agent framework, which employs human-like
strategies for efficient, dynamic, multi-round evaluations using only a few
samples per round, while offering detailed, user-tailored analyses. It offers
four key advantages: 1) efficiency, 2) promptable evaluation tailored to
diverse user needs, 3) explainability beyond single numerical scores, and 4)
scalability across various models and tools. Experiments show that Evaluation
Agent reduces evaluation time to 10% of traditional methods while delivering
comparable results. The Evaluation Agent framework is fully open-sourced to
advance research in visual generative models and their efficient evaluation.
67610b5eb04baaf63514fcc9
Fan Zhang
claimed_verified
2024-12-17T08:03:36.826Z
61f24cbb88b9b5abbe184a85
https://cdn-avatars.huggingface.co/v1/production/uploads/61f24cbb88b9b5abbe184a85/OvcJRU51yI8pdO77NBHLb.jpeg
zhangfan
user
Fan-s
67610b5eb04baaf63514fcca
Shulin Tian
admin_assigned
2024-12-17T09:23:16.414Z
6658d01c6f1a71ba56d6c273
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tc4nZrMuZQLfgt5aVxtH4.jpeg
Tian Shulin
user
shulin16
67610b5eb04baaf63514fccb
Ziqi Huang
admin_assigned
2024-12-17T09:23:04.978Z
60efe7fa0d920bc7805cada5
https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
Ziqi Huang
user
Ziqi
67610b5eb04baaf63514fccc
Yu Qiao
67610b5eb04baaf63514fccd
Ziwei Liu
admin_assigned
2024-12-17T09:22:56.546Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
Evaluation Agent: Efficient and Promptable Evaluation Framework for
  Visual Generative Models
Recent advancements in visual generative models have enabled high-quality
image and video generation, opening diverse applications. However, evaluating
these models often demands sampling hundreds or thousands of images or videos,
making the process computationally expensive, especially for diffusion-based
models with inherently slow sampling. Moreover, existing evaluation methods
rely on rigid pipelines that overlook specific user needs and provide numerical
results without clear explanations. In contrast, humans can quickly form
impressions of a model's capabilities by observing only a few samples. To mimic
this, we propose the Evaluation Agent framework, which employs human-like
strategies for efficient, dynamic, multi-round evaluations using only a few
samples per round, while offering detailed, user-tailored analyses. It offers
four key advantages: 1) efficiency, 2) promptable evaluation tailored to
diverse user needs, 3) explainability beyond single numerical scores, and 4)
scalability across various models and tools. Experiments show that Evaluation
Agent reduces evaluation time to 10% of traditional methods while delivering
comparable results. The Evaluation Agent framework is fully open-sourced to
advance research in visual generative models and their efficient evaluation.
The Evaluation Agent framework efficiently evaluates visual generative models using human-like strategies, reducing time to 10% compared to traditional methods while maintaining quality.
diffusion-based models