Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
Creativity is a fundamental aspect of intelligence, involving the ability to
generate novel and appropriate solutions across diverse contexts. While Large
Language Models (LLMs) have been extensively evaluated for their creative
capabilities, the assessment of Multimodal Large Language Models (MLLMs) in
this domain remains largely unexplored. To address this gap, we introduce
Creation-MMBench, a multimodal benchmark specifically designed to evaluate the
creative capabilities of MLLMs in real-world, image-based tasks. The benchmark
comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous
evaluation, we define instance-specific evaluation criteria for each test case,
guiding the assessment of both general response quality and factual consistency
with visual inputs. Experimental results reveal that current open-source MLLMs
significantly underperform compared to proprietary models in creative tasks.
Furthermore, our analysis demonstrates that visual fine-tuning can negatively
impact the base LLM's creative abilities. Creation-MMBench provides valuable
insights for advancing MLLM creativity and establishes a foundation for future
improvements in multimodal generative intelligence. Full data and evaluation
code is released on https://github.com/open-compass/Creation-MMBench.
67da34a648348387ebac36ff
Xinyu Fang
admin_assigned
2025-03-19T13:17:40.816Z
64f5f8dd9b17cd59c453c57f
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
Xinyu Fang
user
nebulae09
67da34a648348387ebac3700
Zhijian Chen
claimed_verified
2025-03-20T10:46:07.437Z
6726407e98e7ba4c5a2e515f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/iVXMudnRASUhynOAce-1F.png
Zhijian Chen
user
J-Roger
67da34a648348387ebac3701
Kai Lan
67da34a648348387ebac3702
Shengyuan Ding
admin_assigned
2025-03-19T13:18:02.611Z
646cd947da8e99940b6e55cf
https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
Shengyuan Ding
user
ChrisDing1105
67da34a648348387ebac3703
Yingji Liang
67da34a648348387ebac3704
Xiangyu Zhao
claimed_verified
2025-03-19T13:14:48.768Z
6530e62f536dbca918e71c3e
/avatars/efc93bc767e561c6c6d429f65c23382d.svg
Xiangyu Z
user
PhoenixZ
67da34a648348387ebac3705
Farong Wen
67da34a648348387ebac3706
Zicheng Zhang
claimed_verified
2025-04-08T06:57:41.676Z
6526cc6bab4f5d98382f5603
https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
Zicheng Zhang
user
zhangzicheng
67da34a648348387ebac3707
Guofeng Zhang
67da34a648348387ebac3708
Haodong Duan
admin_assigned
2025-03-19T13:18:37.741Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67da34a648348387ebac3709
Kai Chen
67da34a648348387ebac370a
Dahua Lin
admin_assigned
2025-03-19T13:18:43.705Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
Creativity is a fundamental aspect of intelligence, involving the ability to
generate novel and appropriate solutions across diverse contexts. While Large
Language Models (LLMs) have been extensively evaluated for their creative
capabilities, the assessment of Multimodal Large Language Models (MLLMs) in
this domain remains largely unexplored. To address this gap, we introduce
Creation-MMBench, a multimodal benchmark specifically designed to evaluate the
creative capabilities of MLLMs in real-world, image-based tasks. The benchmark
comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous
evaluation, we define instance-specific evaluation criteria for each test case,
guiding the assessment of both general response quality and factual consistency
with visual inputs. Experimental results reveal that current open-source MLLMs
significantly underperform compared to proprietary models in creative tasks.
Furthermore, our analysis demonstrates that visual fine-tuning can negatively
impact the base LLM's creative abilities. Creation-MMBench provides valuable
insights for advancing MLLM creativity and establishes a foundation for future
improvements in multimodal generative intelligence. Full data and evaluation
code is released on https://github.com/open-compass/Creation-MMBench.
Creation-MMBench evaluates the creative capabilities of Multimodal Large Language Models (MLLMs) in image-based real-world tasks, revealing performance gaps and the negative impacts of visual fine-tuning.
Multimodal Large Language Models MLLMs Creation-MMBench instance-specific evaluation criteria visual fine-tuning multimodal generative intelligence