Auto Cherry-Picker: Learning from High-quality Generative Data Driven by
  Language
Diffusion-based models have shown great potential in generating high-quality
images with various layouts, which can benefit downstream perception tasks.
However, a fully automatic layout generation driven only by language and a
suitable metric for measuring multiple generated instances has not been well
explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework
that generates high-quality multi-modal training examples to augment perception
and multi-modal training. Starting with a simple list of natural language
concepts, we prompt large language models (LLMs) to generate a detailed
description and design reasonable layouts. Next, we use an off-the-shelf
text-to-image model to generate multiple images. Then, the generated data are
refined using a comprehensively designed metric to ensure quality. In
particular, we present a new metric, Composite Layout and Image Score (CLIS),
to evaluate the generated images fairly. Our synthetic high-quality examples
boost performance in various scenarios by customizing the initial concept list,
especially in addressing challenges associated with long-tailed distribution
and imbalanced datasets. Experiment results on downstream tasks demonstrate
that Auto Cherry-Picker can significantly improve the performance of existing
models. In addition, we have thoroughly investigated the correlation between
CLIS and performance gains in downstream tasks, and we find that a better CLIS
score results in better performance. This finding shows the potential for
evaluation metrics as the role for various visual perception and MLLM tasks.
Code will be available.
6683d7d517811efbf6741fe9
Yicheng Chen
6683d7d517811efbf6741fea
Xiangtai Li
admin_assigned
2024-07-03T15:19:13.197Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
6683d7d517811efbf6741feb
Yining Li
admin_assigned
2024-07-03T15:19:07.539Z
6570542c91da380dfbdd2d36
/avatars/cf3e7f7c5bf790f93fc6b4cb5d72c915.svg
Yining Li
user
ANTiSOC1AL
6683d7d517811efbf6741fec
Yanhong Zeng
admin_assigned
2024-07-03T15:18:59.658Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
6683d7d517811efbf6741fed
Jianzong Wu
admin_assigned
2024-07-03T15:20:13.413Z
657a6eed1ccc3c2a5ea7b585
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RIQIF-JJdNI0SwJEq_9z7.jpeg
Jianzong Wu
user
jianzongwu
6683d7d517811efbf6741fee
Xiangyu Zhao
claimed_verified
2025-03-19T13:16:10.460Z
6530e62f536dbca918e71c3e
/avatars/efc93bc767e561c6c6d429f65c23382d.svg
Xiangyu Z
user
PhoenixZ
6683d7d517811efbf6741fef
Kai Chen
Auto Cherry-Picker: Learning from High-quality Generative Data Driven by
  Language
Diffusion-based models have shown great potential in generating high-quality
images with various layouts, which can benefit downstream perception tasks.
However, a fully automatic layout generation driven only by language and a
suitable metric for measuring multiple generated instances has not been well
explored. In this work, we present Auto Cherry-Picker (ACP), a novel framework
that generates high-quality multi-modal training examples to augment perception
and multi-modal training. Starting with a simple list of natural language
concepts, we prompt large language models (LLMs) to generate a detailed
description and design reasonable layouts. Next, we use an off-the-shelf
text-to-image model to generate multiple images. Then, the generated data are
refined using a comprehensively designed metric to ensure quality. In
particular, we present a new metric, Composite Layout and Image Score (CLIS),
to evaluate the generated images fairly. Our synthetic high-quality examples
boost performance in various scenarios by customizing the initial concept list,
especially in addressing challenges associated with long-tailed distribution
and imbalanced datasets. Experiment results on downstream tasks demonstrate
that Auto Cherry-Picker can significantly improve the performance of existing
models. In addition, we have thoroughly investigated the correlation between
CLIS and performance gains in downstream tasks, and we find that a better CLIS
score results in better performance. This finding shows the potential for
evaluation metrics as the role for various visual perception and MLLM tasks.
Code will be available.
Auto Cherry-Picker, a framework for generating high-quality multi-modal training examples using language models and a novel metric, enhances performance in perception tasks, particularly addressing long-tailed distributions and imbalanced datasets.
diffusion-based models language models text-to-image CLIS multi-modal visual perception