Bridging the Data Provenance Gap Across Text, Speech and Video
Progress in AI is driven largely by the scale and quality of training data.
Despite this, there is a deficit of empirical analysis examining the attributes
of well-established datasets beyond text. In this work we conduct the largest
and first-of-its-kind longitudinal audit across modalities--popular text,
speech, and video datasets--from their detailed sourcing trends and use
restrictions to their geographical and linguistic representation. Our manual
analysis covers nearly 4000 public datasets between 1990-2024, spanning 608
languages, 798 sources, 659 organizations, and 67 countries. We find that
multimodal machine learning applications have overwhelmingly turned to
web-crawled, synthetic, and social media platforms, such as YouTube, for their
training sets, eclipsing all other sources since 2019. Secondly, tracing the
chain of dataset derivations we find that while less than 33% of datasets are
restrictively licensed, over 80% of the source content in widely-used text,
speech, and video datasets, carry non-commercial restrictions. Finally, counter
to the rising number of languages and geographies represented in public AI
training datasets, our audit demonstrates measures of relative geographical and
multilingual representation have failed to significantly improve their coverage
since 2013. We believe the breadth of our audit enables us to empirically
examine trends in data sourcing, restrictions, and Western-centricity at an
ecosystem-level, and that visibility into these questions are essential to
progress in responsible AI. As a contribution to ongoing improvements in
dataset transparency and responsible use, we release our entire multimodal
audit, allowing practitioners to trace data provenance across text, speech, and
video.
676b6f6e1f5ca46174ac96c8
Shayne Longpre
676b6f6e1f5ca46174ac96c9
Nikhil Singh
claimed_verified
2025-05-29T07:47:22.961Z
6374384c250cf1379bc278f8
/avatars/e436988005ad4f9d99fcafb8bdfade9c.svg
Nikhil Singh
user
nsingh1
676b6f6e1f5ca46174ac96ca
Manuel Cherep
676b6f6e1f5ca46174ac96cb
Kushagra Tiwary
676b6f6e1f5ca46174ac96cc
Joanna Materzynska
676b6f6e1f5ca46174ac96cd
William Brannon
claimed_verified
2025-02-06T14:15:29.452Z
641bdd25a63c4e8062387b6a
https://cdn-avatars.huggingface.co/v1/production/uploads/641bdd25a63c4e8062387b6a/dFanBhG6_NVqpRNB-jjC4.png
William Brannon
user
wwbrannon
676b6f6e1f5ca46174ac96ce
Robert Mahari
676b6f6e1f5ca46174ac96cf
Manan Dey
676b6f6e1f5ca46174ac96d0
Mohammed Hamdy
claimed_verified
2024-12-30T19:33:07.482Z
62645f88c39850dc093d6105
https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
Mohammed Hamdy
user
mmhamdy
676b6f6e1f5ca46174ac96d1
Nayan Saxena
676b6f6e1f5ca46174ac96d2
Ahmad Mustafa Anis
claimed_verified
2024-12-30T19:33:00.926Z
6246908d8031dcfa9ef6d80b
https://cdn-avatars.huggingface.co/v1/production/uploads/6246908d8031dcfa9ef6d80b/hVdURjUl1RS2MZf4qOhvI.jpeg
Ahmad Mustafa Anis
user
AhmadMustafa
676b6f6e1f5ca46174ac96d3
Emad A. Alghamdi
676b6f6e1f5ca46174ac96d4
Vu Minh Chien
claimed_verified
2025-02-12T09:19:03.796Z
60535c9d10aba34e3b6a2ef7
https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
vumichien
user
vumichien
676b6f6e1f5ca46174ac96d5
Naana Obeng-Marnu
676b6f6e1f5ca46174ac96d6
Da Yin
676b6f6e1f5ca46174ac96d7
Kun Qian
676b6f6e1f5ca46174ac96d8
Yizhi Li
claimed_verified
2025-07-25T08:40:07.901Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
676b6f6e1f5ca46174ac96d9
Minnie Liang
676b6f6e1f5ca46174ac96da
An Dinh
676b6f6e1f5ca46174ac96db
Shrestha Mohanty
676b6f6e1f5ca46174ac96dc
Deividas Mataciunas
claimed_verified
2024-12-30T19:33:05.129Z
6040a00558b78f3a0047c23a
https://cdn-avatars.huggingface.co/v1/production/uploads/6040a00558b78f3a0047c23a/_BsyJoaCBO3r6GnfSgnIA.jpeg
David Mataciunas
user
DeividasM
676b6f6e1f5ca46174ac96dd
Tobin South
676b6f6e1f5ca46174ac96de
Jianguo Zhang
676b6f6e1f5ca46174ac96df
Ariel N. Lee
claimed_verified
2025-02-05T16:54:01.989Z
638bcfa91987d67b340e6c1c
https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
Ariel N. Lee
user
arielnlee
676b6f6e1f5ca46174ac96e0
Campbell S. Lund
676b6f6e1f5ca46174ac96e1
Christopher Klamm
676b6f6e1f5ca46174ac96e2
Damien Sileo
676b6f6e1f5ca46174ac96e3
Diganta Misra
676b6f6e1f5ca46174ac96e4
Enrico Shippole
claimed_verified
2025-06-23T08:34:23.309Z
6276ba3c2d26ac639e5a2b01
https://cdn-avatars.huggingface.co/v1/production/uploads/6276ba3c2d26ac639e5a2b01/ULDyT9ijOye7rfXqTWFVG.jpeg
Enrico Shippole
user
conceptofmind
676b6f6e1f5ca46174ac96e5
Kevin Klyman
676b6f6e1f5ca46174ac96e6
Lester JV Miranda
claimed_verified
2025-01-06T08:00:50.985Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
676b6f6e1f5ca46174ac96e7
Niklas Muennighoff
676b6f6e1f5ca46174ac96e8
Seonghyeon Ye
676b6f6e1f5ca46174ac96e9
Seungone Kim
claimed_verified
2024-12-30T19:33:02.972Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
676b6f6e1f5ca46174ac96ea
Vipul Gupta
676b6f6e1f5ca46174ac96eb
Vivek Sharma
676b6f6e1f5ca46174ac96ec
Xuhui Zhou
676b6f6e1f5ca46174ac96ed
Caiming Xiong
676b6f6e1f5ca46174ac96ee
Luis Villa
676b6f6e1f5ca46174ac96ef
Stella Biderman
claimed_verified
2025-06-07T05:50:33.769Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
676b6f6e1f5ca46174ac96f0
Alex Pentland
676b6f6e1f5ca46174ac96f1
Sara Hooker
claimed_verified
2025-01-20T09:30:34.082Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
676b6f6e1f5ca46174ac96f2
Jad Kabbara
Bridging the Data Provenance Gap Across Text, Speech and Video
Progress in AI is driven largely by the scale and quality of training data.
Despite this, there is a deficit of empirical analysis examining the attributes
of well-established datasets beyond text. In this work we conduct the largest
and first-of-its-kind longitudinal audit across modalities--popular text,
speech, and video datasets--from their detailed sourcing trends and use
restrictions to their geographical and linguistic representation. Our manual
analysis covers nearly 4000 public datasets between 1990-2024, spanning 608
languages, 798 sources, 659 organizations, and 67 countries. We find that
multimodal machine learning applications have overwhelmingly turned to
web-crawled, synthetic, and social media platforms, such as YouTube, for their
training sets, eclipsing all other sources since 2019. Secondly, tracing the
chain of dataset derivations we find that while less than 33% of datasets are
restrictively licensed, over 80% of the source content in widely-used text,
speech, and video datasets, carry non-commercial restrictions. Finally, counter
to the rising number of languages and geographies represented in public AI
training datasets, our audit demonstrates measures of relative geographical and
multilingual representation have failed to significantly improve their coverage
since 2013. We believe the breadth of our audit enables us to empirically
examine trends in data sourcing, restrictions, and Western-centricity at an
ecosystem-level, and that visibility into these questions are essential to
progress in responsible AI. As a contribution to ongoing improvements in
dataset transparency and responsible use, we release our entire multimodal
audit, allowing practitioners to trace data provenance across text, speech, and
video.
multimodal machine learning applications web-crawled synthetic social media platforms YouTube dataset derivations restrictively licensed non-commercial restrictions geographical representation multilingual representation Western-centricity data sourcing dataset transparency responsible use data provenance