CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large
  Language Models
We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered
for superior data quality and diverse human-like reasoning trajectory. CCI4.0
occupies roughly 35 TB of disk space and comprises two sub-datasets:
CCI4.0-M2-Base and CCI4.0-M2-CoT. CCI4.0-M2-Base combines a 5.2 TB carefully
curated Chinese web corpus, a 22.5 TB English subset from Nemotron-CC, and
diverse sources from math, wiki, arxiv, and code. Although these data are
mostly sourced from well-processed datasets, the quality standards of various
domains are dynamic and require extensive expert experience and labor to
process. So, we propose a novel pipeline justifying data quality mainly based
on models through two-stage deduplication, multiclassifier quality scoring, and
domain-aware fluency filtering. We extract 4.5 billion pieces of
CoT(Chain-of-Thought) templates, named CCI4.0-M2-CoT. Differing from the
distillation of CoT from larger models, our proposed staged CoT extraction
exemplifies diverse reasoning patterns and significantly decreases the
possibility of hallucination. Empirical evaluations demonstrate that LLMs
pre-trained in CCI4.0 benefit from cleaner, more reliable training signals,
yielding consistent improvements in downstream tasks, especially in math and
code reflection tasks. Our results underscore the critical role of rigorous
data curation and human thinking templates in advancing LLM performance,
shedding some light on automatically processing pretraining corpora.
68478a493ec10bdd8ab4dd90
Guang Liu
claimed_verified
2025-06-10T09:19:47.891Z
632c234f42c386ebd2710434
https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
Guang Liu
user
ZacLiu
68478a493ec10bdd8ab4dd91
Liangdong Wang
claimed_verified
2025-06-10T09:19:50.398Z
63a11ce02fabbbb899a01d58
/avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
ldwang
user
ldwang
68478a493ec10bdd8ab4dd92
Jijie Li
68478a493ec10bdd8ab4dd93
Yang Yu
68478a493ec10bdd8ab4dd94
Yao Xu
68478a493ec10bdd8ab4dd95
Jiabei Chen
68478a493ec10bdd8ab4dd96
Yu Bai
68478a493ec10bdd8ab4dd97
Feng Liao
68478a493ec10bdd8ab4dd98
Yonghua Lin
admin_assigned
2025-06-10T09:39:44.976Z
629aa3155ab4232a3fe0893e
/avatars/cf2d4a9295b5da9e2e4d2278bbb36040.svg
Yonghua Lin
user
Yonghua
CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large
  Language Models
We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered
for superior data quality and diverse human-like reasoning trajectory. CCI4.0
occupies roughly 35 TB of disk space and comprises two sub-datasets:
CCI4.0-M2-Base and CCI4.0-M2-CoT. CCI4.0-M2-Base combines a 5.2 TB carefully
curated Chinese web corpus, a 22.5 TB English subset from Nemotron-CC, and
diverse sources from math, wiki, arxiv, and code. Although these data are
mostly sourced from well-processed datasets, the quality standards of various
domains are dynamic and require extensive expert experience and labor to
process. So, we propose a novel pipeline justifying data quality mainly based
on models through two-stage deduplication, multiclassifier quality scoring, and
domain-aware fluency filtering. We extract 4.5 billion pieces of
CoT(Chain-of-Thought) templates, named CCI4.0-M2-CoT. Differing from the
distillation of CoT from larger models, our proposed staged CoT extraction
exemplifies diverse reasoning patterns and significantly decreases the
possibility of hallucination. Empirical evaluations demonstrate that LLMs
pre-trained in CCI4.0 benefit from cleaner, more reliable training signals,
yielding consistent improvements in downstream tasks, especially in math and
code reflection tasks. Our results underscore the critical role of rigorous
data curation and human thinking templates in advancing LLM performance,
shedding some light on automatically processing pretraining corpora.
A large-scale bilingual pre-training dataset, CCI4.0, enhances data quality and diverse reasoning patterns for language models, leading to improved performance in downstream tasks like math and code reflection.
pre-training dataset bilingual pre-training data quality reasoning trajectory deduplication multiclassifier quality scoring domain-aware fluency filtering Chain-of-Thought CoT extraction language models LLMs downstream tasks math tasks code reflection tasks data curation human thinking templates