Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for
  Cartoon Captioning
We present a novel multimodal preference dataset for creative tasks,
consisting of over 250 million human ratings on more than 2.2 million captions,
collected through crowdsourcing rating data for The New Yorker's weekly cartoon
caption contest over the past eight years. This unique dataset supports the
development and evaluation of multimodal large language models and
preference-based fine-tuning algorithms for humorous caption generation. We
propose novel benchmarks for judging the quality of model-generated captions,
utilizing both GPT4 and human judgments to establish ranking-based evaluation
strategies. Our experimental results highlight the limitations of current
fine-tuning methods, such as RLHF and DPO, when applied to creative tasks.
Furthermore, we demonstrate that even state-of-the-art models like GPT4 and
Claude currently underperform top human contestants in generating humorous
captions. As we conclude this extensive data collection effort, we release the
entire preference dataset to the research community, fostering further
advancements in AI humor generation and evaluation.
6671b1468b5e22ce4f2ca4ad
Jifan Zhang
claimed_verified
2024-06-18T21:26:52.590Z
662a3a061bbfbf02afb1db90
/avatars/f2cdeddb2810a3d9792b8c69c9e48ade.svg
Jifan Zhang
user
jifanz
6671b1468b5e22ce4f2ca4ae
Lalit Jain
6671b1468b5e22ce4f2ca4af
Yang Guo
6671b1468b5e22ce4f2ca4b0
Jiayi Chen
6671b1468b5e22ce4f2ca4b1
Kuan Lok Zhou
6671b1468b5e22ce4f2ca4b2
Siddharth Suresh
6671b1468b5e22ce4f2ca4b3
Andrew Wagenmaker
6671b1468b5e22ce4f2ca4b4
Scott Sievert
6671b1468b5e22ce4f2ca4b5
Timothy Rogers
6671b1468b5e22ce4f2ca4b6
Kevin Jamieson
6671b1468b5e22ce4f2ca4b7
Robert Mankoff
6671b1468b5e22ce4f2ca4b8
Robert Nowak
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for
  Cartoon Captioning
We present a novel multimodal preference dataset for creative tasks,
consisting of over 250 million human ratings on more than 2.2 million captions,
collected through crowdsourcing rating data for The New Yorker's weekly cartoon
caption contest over the past eight years. This unique dataset supports the
development and evaluation of multimodal large language models and
preference-based fine-tuning algorithms for humorous caption generation. We
propose novel benchmarks for judging the quality of model-generated captions,
utilizing both GPT4 and human judgments to establish ranking-based evaluation
strategies. Our experimental results highlight the limitations of current
fine-tuning methods, such as RLHF and DPO, when applied to creative tasks.
Furthermore, we demonstrate that even state-of-the-art models like GPT4 and
Claude currently underperform top human contestants in generating humorous
captions. As we conclude this extensive data collection effort, we release the
entire preference dataset to the research community, fostering further
advancements in AI humor generation and evaluation.
The dataset supports evaluating multimodal language models and preference-based fine-tuning for humor generation, revealing current models' limitations compared to humans.
multimodal preference dataset creative tasks human ratings captions crowdsourcing The New Yorker's weekly cartoon multimodal large language models preference-based fine-tuning humorous caption generation GPT4 human judgments ranking-based evaluation RLHF DPO AI humor generation