Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Liwei Jiang, Chai Yuanjun, Margaret Li, Mickel Liu, Raymond Fok, Maarten Sap, Yulia Tsvetkov, Nouha Dziri, Yejin Choi
Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet, scalable methods for evaluating LM output diversity remain limited—especially beyond narrow tasks like random number generation or stylized prompts. To address this gap, we introduce InfiniteChats, a large-scale dataset of 26,000 diverse, real-world open-ended user queries, along with the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising six top-level categories and 17 subcategories. These queries admit a wide range of plausible answers with no single ground truth. Using InfiniteChats, we present a large-scale analysis of mode collapse in LMs, manifested as redundant outputs even for inherently open-ended queries. Our study reveals a pronounced "Artificial Hivemind" effect in open-ended generation, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and (2) inter-model homogeneity, where different models produce strikingly similar outputs.InfiniteChats also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables fine-grained analysis of distributional preferences across annotators. Our findings show that state-of-the-art LMs, reward models, and LM judges align less with human ratings when annotators disagree or when responses are of similar quality. Overall, InfiniteChats offers the first large-scale resource for systematically studying open-endedness in LM queries, revealing critical insights to guide future research and mitigate long-term AI safety risks posed by the Artificial Hivemind.