MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality
  Instruction and Reasoning Data Built from Permissive-First Text Sources
We present MixtureVitae, an open-access pretraining corpus built to minimize
legal risk while providing strong model performance. MixtureVitae follows a
risk-mitigated sourcing strategy that combines public-domain and permissively
licensed text (e.g., CC-BY/Apache) with carefully justified low-risk additions
(e.g., government works and EU TDM-eligible sources), alongside targeted
instruction, reasoning and synthetic data with documented provenance. We detail
a transparent, multi-stage pipeline for license-aware filtering, safety and
quality screening, and domain-aware mixing, and we release the dataset and
curation recipes to support reproducible research. In controlled experiments
using the open-sci-ref training protocol (fixed architectures at
130M/400M/1.3B/1.7B parameters; training budgets of 50B and 300B tokens),
models trained on MixtureVitae consistently outperform other permissive
datasets across a suite of standard benchmarks, and at the 1.7B/300B setting
they surpass FineWeb-Edu and approach DCLM in the later stages of training.
Performance is particularly strong on math/code and competitive on QA tasks.
These results demonstrate that permissive-first, risk-mitigated data provides a
practical and legally mitigated foundation for training capable LLMs, reducing
reliance on indiscriminate web scraping without sacrificing competitiveness.
Code: https://github.com/ontocord/mixturevitae
68dd24dee795697646c0a134
Huu Nguyen
claimed_verified
2025-10-03T12:54:52.084Z
5fc6879e1c5ee87b1164876d
https://cdn-avatars.huggingface.co/v1/production/uploads/5fc6879e1c5ee87b1164876d/Tjnm_lv0Bq0gPbFOTDH6E.jpeg
Huu Nguyen
user
huu-ontocord
68dd24dee795697646c0a135
Victor May
68dd24dee795697646c0a136
Harsh Raj
68dd24dee795697646c0a137
Marianna Nezhurina
68dd24dee795697646c0a138
Yishan Wang
68dd24dee795697646c0a139
Yanqi Luo
68dd24dee795697646c0a13a
Minh Chien Vu
claimed_verified
2025-10-14T07:32:50.463Z
60535c9d10aba34e3b6a2ef7
https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
vumichien
user
vumichien
68dd24dee795697646c0a13b
Taishi Nakamura
claimed_verified
2025-10-02T03:08:37.727Z
6308c49c454dc257521bc7f9
https://cdn-avatars.huggingface.co/v1/production/uploads/6308c49c454dc257521bc7f9/UWUS6OPa6OpVu1T0gd-wJ.jpeg
Taishi
user
Taishi-N324
68dd24dee795697646c0a13c
Ken Tsui
claimed_verified
2025-10-05T12:47:35.774Z
60e50ce5350d181892d5a636
https://cdn-avatars.huggingface.co/v1/production/uploads/60e50ce5350d181892d5a636/qjYikZtE-2kjub-6XG5Kn.jpeg
Ken Tsui
user
kenhktsui
68dd24dee795697646c0a13d
Van Khue Nguyen
claimed_verified
2025-10-11T14:05:55.673Z
67e2556c54c126c06427ff3c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/oTI9SmXTH3-t401zIk_--.jpeg
Nguyễn Văn Khuê
user
TieuDaoChanNhan
68dd24dee795697646c0a13e
David Salinas
68dd24dee795697646c0a13f
Aleksandra Krasnodębska
68dd24dee795697646c0a140
Christoph Schuhmann
68dd24dee795697646c0a141
Mats Leon Richter
68dd24dee795697646c0a142
Xuan-Son
68dd24dee795697646c0a143
Vu
68dd24dee795697646c0a144
Jenia Jitsev
MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality
  Instruction and Reasoning Data Built from Permissive-First Text Sources
We present MixtureVitae, an open-access pretraining corpus built to minimize
legal risk while providing strong model performance. MixtureVitae follows a
risk-mitigated sourcing strategy that combines public-domain and permissively
licensed text (e.g., CC-BY/Apache) with carefully justified low-risk additions
(e.g., government works and EU TDM-eligible sources), alongside targeted
instruction, reasoning and synthetic data with documented provenance. We detail
a transparent, multi-stage pipeline for license-aware filtering, safety and
quality screening, and domain-aware mixing, and we release the dataset and
curation recipes to support reproducible research. In controlled experiments
using the open-sci-ref training protocol (fixed architectures at
130M/400M/1.3B/1.7B parameters; training budgets of 50B and 300B tokens),
models trained on MixtureVitae consistently outperform other permissive
datasets across a suite of standard benchmarks, and at the 1.7B/300B setting
they surpass FineWeb-Edu and approach DCLM in the later stages of training.
Performance is particularly strong on math/code and competitive on QA tasks.
These results demonstrate that permissive-first, risk-mitigated data provides a
practical and legally mitigated foundation for training capable LLMs, reducing
reliance on indiscriminate web scraping without sacrificing competitiveness.
Code: https://github.com/ontocord/mixturevitae
MixtureVitae is a pretraining corpus that combines public-domain and permissively licensed text with low-risk additions, achieving strong model performance across benchmarks while minimizing legal risk.
pretraining corpus risk-mitigated sourcing strategy public-domain permissively licensed text CC-BY Apache government works EU TDM-eligible sources targeted instruction reasoning synthetic data license-aware filtering safety screening quality screening domain-aware mixing open-sci-ref training protocol LLMs FineWeb-Edu DCLM math/code QA tasks