Consent in Crisis: The Rapid Decline of the AI Data Commons
General-purpose artificial intelligence (AI) systems are built on massive
swathes of public web data, assembled into corpora such as C4, RefinedWeb, and
Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit
of the consent protocols for the web domains underlying AI training corpora.
Our audit of 14,000 web domains provides an expansive view of crawlable web
data and how consent preferences to use it are changing over time. We observe a
proliferation of AI-specific clauses to limit use, acute differences in
restrictions on AI developers, as well as general inconsistencies between
websites' expressed intentions in their Terms of Service and their robots.txt.
We diagnose these as symptoms of ineffective web protocols, not designed to
cope with the widespread re-purposing of the internet for AI. Our longitudinal
analyses show that in a single year (2023-2024) there has been a rapid
crescendo of data restrictions from web sources, rendering ~5%+ of all tokens
in C4, or 28%+ of the most actively maintained, critical sources in C4, fully
restricted from use. For Terms of Service crawling restrictions, a full 45% of
C4 is now restricted. If respected or enforced, these restrictions are rapidly
biasing the diversity, freshness, and scaling laws for general-purpose AI
systems. We hope to illustrate the emerging crisis in data consent, foreclosing
much of the open web, not only for commercial AI, but non-commercial AI and
academic purposes.
669f84f13e1b77c317b64345
Shayne Longpre
669f84f13e1b77c317b64346
Robert Mahari
claimed_verified
2024-09-25T07:35:25.946Z
650e085c37698373cdda7a6d
/avatars/8fae18709d51dd86aa6eddcfb2abeae8.svg
Robert Mahari
user
rmahari
669f84f13e1b77c317b64347
Ariel Lee
claimed_verified
2024-07-29T07:06:14.077Z
638bcfa91987d67b340e6c1c
https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
Ariel N. Lee
user
arielnlee
669f84f13e1b77c317b64348
Campbell Lund
669f84f13e1b77c317b64349
Hamidah Oderinwale
claimed_verified
2025-08-13T07:23:40.597Z
679b0f84009d1e01589437a5
https://cdn-avatars.huggingface.co/v1/production/uploads/679b0f84009d1e01589437a5/6w6BfIPPdj9SoedZRgN9J.png
Hamidah Oderinwale
user
midah
669f84f13e1b77c317b6434a
William Brannon
claimed_verified
2024-11-15T09:46:19.380Z
641bdd25a63c4e8062387b6a
https://cdn-avatars.huggingface.co/v1/production/uploads/641bdd25a63c4e8062387b6a/dFanBhG6_NVqpRNB-jjC4.png
William Brannon
user
wwbrannon
669f84f13e1b77c317b6434b
Nayan Saxena
669f84f13e1b77c317b6434c
Naana Obeng-Marnu
669f84f13e1b77c317b6434d
Tobin South
669f84f13e1b77c317b6434e
Cole Hunter
669f84f13e1b77c317b6434f
Kevin Klyman
669f84f13e1b77c317b64350
Christopher Klamm
669f84f13e1b77c317b64351
Hailey Schoelkopf
669f84f13e1b77c317b64352
Nikhil Singh
claimed_verified
2025-05-29T07:47:45.558Z
6374384c250cf1379bc278f8
/avatars/e436988005ad4f9d99fcafb8bdfade9c.svg
Nikhil Singh
user
nsingh1
669f84f13e1b77c317b64353
Manuel Cherep
669f84f13e1b77c317b64354
Ahmad Anis
claimed_verified
2024-09-30T08:08:10.262Z
6246908d8031dcfa9ef6d80b
https://cdn-avatars.huggingface.co/v1/production/uploads/6246908d8031dcfa9ef6d80b/hVdURjUl1RS2MZf4qOhvI.jpeg
Ahmad Mustafa Anis
user
AhmadMustafa
669f84f13e1b77c317b64355
An Dinh
669f84f13e1b77c317b64356
Caroline Chitongo
669f84f13e1b77c317b64357
Da Yin
669f84f13e1b77c317b64358
Damien Sileo
claimed_verified
2024-07-29T07:06:15.859Z
5fc0bcb41160c47d1d43856b
https://cdn-avatars.huggingface.co/v1/production/uploads/5fc0bcb41160c47d1d43856b/AHCEW4TfTdyjNBx-V_F5A.png
Damien Sileo
user
sileod
669f84f13e1b77c317b64359
Deividas Mataciunas
claimed_verified
2024-07-29T14:58:08.268Z
6040a00558b78f3a0047c23a
https://cdn-avatars.huggingface.co/v1/production/uploads/6040a00558b78f3a0047c23a/_BsyJoaCBO3r6GnfSgnIA.jpeg
David Mataciunas
user
DeividasM
669f84f13e1b77c317b6435a
Diganta Misra
669f84f13e1b77c317b6435b
Emad Alghamdi
669f84f13e1b77c317b6435c
Enrico Shippole
claimed_verified
2024-07-24T07:27:13.474Z
6276ba3c2d26ac639e5a2b01
https://cdn-avatars.huggingface.co/v1/production/uploads/6276ba3c2d26ac639e5a2b01/ULDyT9ijOye7rfXqTWFVG.jpeg
Enrico Shippole
user
conceptofmind
669f84f13e1b77c317b6435d
Jianguo Zhang
669f84f13e1b77c317b6435e
Joanna Materzynska
669f84f13e1b77c317b6435f
Kun Qian
669f84f13e1b77c317b64360
Kush Tiwary
669f84f13e1b77c317b64361
Lester Miranda
claimed_verified
2024-08-13T07:39:45.367Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
669f84f13e1b77c317b64362
Manan Dey
669f84f13e1b77c317b64363
Minnie Liang
669f84f13e1b77c317b64364
Mohammed Hamdy
claimed_verified
2024-07-23T11:43:16.696Z
62645f88c39850dc093d6105
https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
Mohammed Hamdy
user
mmhamdy
669f84f13e1b77c317b64365
Niklas Muennighoff
669f84f13e1b77c317b64366
Seonghyeon Ye
669f84f13e1b77c317b64367
Seungone Kim
claimed_verified
2024-09-02T07:19:11.856Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
669f84f13e1b77c317b64368
Shrestha Mohanty
669f84f13e1b77c317b64369
Vipul Gupta
669f84f13e1b77c317b6436a
Vivek Sharma
669f84f13e1b77c317b6436b
Vu Minh Chien
claimed_verified
2024-07-26T07:40:55.488Z
60535c9d10aba34e3b6a2ef7
https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
vumichien
user
vumichien
669f84f13e1b77c317b6436c
Xuhui Zhou
669f84f13e1b77c317b6436d
Yizhi Li
claimed_verified
2025-07-25T08:40:34.962Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
669f84f13e1b77c317b6436e
Caiming Xiong
669f84f13e1b77c317b6436f
Luis Villa
669f84f13e1b77c317b64370
Stella Biderman
claimed_verified
2025-06-07T05:51:07.896Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
669f84f13e1b77c317b64371
Hanlin Li
669f84f13e1b77c317b64372
Daphne Ippolito
669f84f13e1b77c317b64373
Sara Hooker
claimed_verified
2024-09-03T07:48:51.508Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
669f84f13e1b77c317b64374
Jad Kabbara
669f84f13e1b77c317b64375
Sandy Pentland
Consent in Crisis: The Rapid Decline of the AI Data Commons
General-purpose artificial intelligence (AI) systems are built on massive
swathes of public web data, assembled into corpora such as C4, RefinedWeb, and
Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit
of the consent protocols for the web domains underlying AI training corpora.
Our audit of 14,000 web domains provides an expansive view of crawlable web
data and how consent preferences to use it are changing over time. We observe a
proliferation of AI-specific clauses to limit use, acute differences in
restrictions on AI developers, as well as general inconsistencies between
websites' expressed intentions in their Terms of Service and their robots.txt.
We diagnose these as symptoms of ineffective web protocols, not designed to
cope with the widespread re-purposing of the internet for AI. Our longitudinal
analyses show that in a single year (2023-2024) there has been a rapid
crescendo of data restrictions from web sources, rendering ~5%+ of all tokens
in C4, or 28%+ of the most actively maintained, critical sources in C4, fully
restricted from use. For Terms of Service crawling restrictions, a full 45% of
C4 is now restricted. If respected or enforced, these restrictions are rapidly
biasing the diversity, freshness, and scaling laws for general-purpose AI
systems. We hope to illustrate the emerging crisis in data consent, foreclosing
much of the open web, not only for commercial AI, but non-commercial AI and
academic purposes.
The audit of web domains used for training AI systems reveals a growing inconsistency and restriction of data use, posing a significant threat to the diversity and availability of open web data for AI.
Terms of Service robots.txt data restrictions AI training corpora consent protocols data consent crisis