RefineX: Learning to Refine Pre-training Data at Scale from
  Expert-Guided Programs
The foundational capabilities of large language models (LLMs) are deeply
influenced by the quality of their pre-training corpora. However, enhancing
data quality at scale remains a significant challenge, primarily due to the
trade-off between refinement effectiveness and processing efficiency. While
rule-based filtering remains the dominant paradigm, it typically operates at
the document level and lacks the granularity needed to refine specific content
within documents. Inspired by emerging work such as ProX, we propose
RefineX, a novel framework for large-scale, surgical refinement of
pre-training data through programmatic editing tasks. RefineX enables efficient
and fine-grained data refinement while reliably preserving the diversity and
naturalness of raw text. The core strength of RefineX lies in distilling
high-quality, expert-guided end-to-end refinement results into minimal
edit-based deletion programs. This high-precision distillation pipeline is used
to train an efficient and reliable refine model that can systematically improve
every instance in the corpus at scale. We evaluate RefineX across from-scratch
pre-training at multiple model scales and find that it consistently outperforms
models trained on raw, filtered, or alternatively refined data across diverse
downstream tasks. On the 750M model, RefineX yields 2.6%-7.2% average gains on
lighteval tasks, and achieves comparable performance using significantly fewer
training tokens. Further analysis shows that RefineX reliably enhances text
quality with both high efficiency and precision, outperforming prior approaches
such as end-to-end generation and Prox-C. These results position RefineX as a
scalable, effective, and reliable solution for optimizing pre-training data in
modern LLM pipelines.
686c86ff364e2ad167eb53a8
Baolong Bi
claimed_verified
2025-07-29T07:10:53.152Z
642577e06d0f0f5f1dc68904
/avatars/3df7cfcf9ca6cd9c9e9b10a73f4efc35.svg
Bibaolong
user
Bibaolong
686c86ff364e2ad167eb53a9
Shenghua Liu
686c86ff364e2ad167eb53aa
Xingzhang Ren
686c86ff364e2ad167eb53ab
Dayiheng Liu
686c86ff364e2ad167eb53ac
Junyang Lin
686c86ff364e2ad167eb53ad
Yiwei Wang
686c86ff364e2ad167eb53ae
Lingrui Mei
claimed_verified
2025-07-08T09:05:44.909Z
63120517ae8896941da4c5da
/avatars/10e1be026035f3e24225e6782a710083.svg
Lingrui Mei
user
Chevalier
686c86ff364e2ad167eb53af
Junfeng Fang
686c86ff364e2ad167eb53b0
Jiafeng Guo
686c86ff364e2ad167eb53b1
Xueqi Cheng
RefineX: Learning to Refine Pre-training Data at Scale from
  Expert-Guided Programs
The foundational capabilities of large language models (LLMs) are deeply
influenced by the quality of their pre-training corpora. However, enhancing
data quality at scale remains a significant challenge, primarily due to the
trade-off between refinement effectiveness and processing efficiency. While
rule-based filtering remains the dominant paradigm, it typically operates at
the document level and lacks the granularity needed to refine specific content
within documents. Inspired by emerging work such as ProX, we propose
RefineX, a novel framework for large-scale, surgical refinement of
pre-training data through programmatic editing tasks. RefineX enables efficient
and fine-grained data refinement while reliably preserving the diversity and
naturalness of raw text. The core strength of RefineX lies in distilling
high-quality, expert-guided end-to-end refinement results into minimal
edit-based deletion programs. This high-precision distillation pipeline is used
to train an efficient and reliable refine model that can systematically improve
every instance in the corpus at scale. We evaluate RefineX across from-scratch
pre-training at multiple model scales and find that it consistently outperforms
models trained on raw, filtered, or alternatively refined data across diverse
downstream tasks. On the 750M model, RefineX yields 2.6%-7.2% average gains on
lighteval tasks, and achieves comparable performance using significantly fewer
training tokens. Further analysis shows that RefineX reliably enhances text
quality with both high efficiency and precision, outperforming prior approaches
such as end-to-end generation and Prox-C. These results position RefineX as a
scalable, effective, and reliable solution for optimizing pre-training data in
modern LLM pipelines.
RefineX is a framework that uses programmatic editing tasks to efficiently refine large-scale pre-training data for LLMs, improving text quality and performance across various tasks.
large language models pre-training corpora data quality programmatic editing tasks refine model end-to-end refinement text quality lighteval tasks training tokens ProX Prox-C