Evaluating Text Creativity across Diverse Domains: A Dataset and Large
  Language Model Evaluator
Creativity evaluation remains a challenging frontier for large language
models (LLMs). Current evaluations heavily rely on inefficient and costly human
judgments, hindering progress in enhancing machine creativity. While automated
methods exist, ranging from psychological testing to heuristic- or
prompting-based approaches, they often lack generalizability or alignment with
human judgment. To address these issues, in this paper, we propose a novel
pairwise-comparison framework for assessing textual creativity, leveraging
shared contextual instructions to improve evaluation consistency. We introduce
CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic
creative instruction-response pairs spanning diverse open-domain tasks. Through
training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval
demonstrates remarkable superiority over existing methods in alignment with
human judgments. Experimental results underscore the indispensable significance
of integrating both human-generated and synthetic data in training highly
robust evaluators, and showcase the practical utility of CrEval in boosting the
creativity of LLMs. We will release all data, code, and models publicly soon to
support further research.
68392fc4b6280677f75e6194
Qian Cao
claimed_verified
2025-05-30T06:53:24.101Z
5fbdf878485ef14d9a960f4d
https://cdn-avatars.huggingface.co/v1/production/uploads/1634724388998-5fbdf878485ef14d9a960f4d.jpeg
Qian Cao
user
Aman
68392fc4b6280677f75e6195
Xiting Wang
68392fc4b6280677f75e6196
Yuzhuo Yuan
68392fc4b6280677f75e6197
Yahui Liu
68392fc4b6280677f75e6198
Fang Luo
68392fc4b6280677f75e6199
Ruihua Song
Evaluating Text Creativity across Diverse Domains: A Dataset and Large
  Language Model Evaluator
Creativity evaluation remains a challenging frontier for large language
models (LLMs). Current evaluations heavily rely on inefficient and costly human
judgments, hindering progress in enhancing machine creativity. While automated
methods exist, ranging from psychological testing to heuristic- or
prompting-based approaches, they often lack generalizability or alignment with
human judgment. To address these issues, in this paper, we propose a novel
pairwise-comparison framework for assessing textual creativity, leveraging
shared contextual instructions to improve evaluation consistency. We introduce
CreataSet, a large-scale dataset with 100K+ human-level and 1M+ synthetic
creative instruction-response pairs spanning diverse open-domain tasks. Through
training on CreataSet, we develop an LLM-based evaluator named CrEval. CrEval
demonstrates remarkable superiority over existing methods in alignment with
human judgments. Experimental results underscore the indispensable significance
of integrating both human-generated and synthetic data in training highly
robust evaluators, and showcase the practical utility of CrEval in boosting the
creativity of LLMs. We will release all data, code, and models publicly soon to
support further research.
A novel pairwise-comparison framework using CreataSet dataset trains CrEval, an LLM-based evaluator that significantly improves the assessment of textual creativity aligned with human judgments.
large language models LLMs automated methods heuristic-based approaches prompting-based approaches pairwise-comparison framework shared contextual instructions CreataSet synthetic creative instruction-response pairs open-domain tasks human-level instructions CrEval human judgments robust evaluators creativity enhancement