Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture
  without Training
We introduce~Domain2Vec, a novel approach that decomposes any
dataset into a linear combination of several meta-domains, a new concept
designed to capture the key underlying features of datasets.
Domain2Vec maintains a vocabulary of meta-domains and uses a
classifier to decompose any given dataset into a domain vector that corresponds
to a distribution over this vocabulary. These domain vectors enable the
identification of the optimal data mixture for language model (LM) pretraining
in a training-free manner under the \textbf{Distribution
Alignment Assumption} (DA^{2}), which suggests that when
the data distributions of the training set and the validation set are better
aligned, a lower validation loss is achieved. Moreover, Domain2vec can
be seamlessly integrated into previous works to model the relationship between
domain vectors and LM performance, greatly enhancing the efficiency and
scalability of previous methods. Extensive experiments demonstrate that
Domain2Vec helps find the data mixture that enhances downstream task
performance with minimal computational overhead. Specifically,
Domain2Vec achieves the same validation loss on Pile-CC using only
51.5% of the computation required when training on the original mixture of
The Pile dataset. Under equivalent compute budget, Domain2Vec improves
downstream performance by an average of 2.83%.
684b96403b733ba33368703a
Mozhi Zhang
claimed_verified
2025-06-13T07:39:33.791Z
65e808ed7c10574cc3f8e363
/avatars/ed10759d354e271bfc15afd946b66b4a.svg
zhangmozhi
user
mzzhang
684b96403b733ba33368703b
Howe Tissue
claimed_verified
2025-06-13T07:39:31.965Z
6718fc605e14ff6b94a7109f
https://cdn-avatars.huggingface.co/v1/production/uploads/6718fc605e14ff6b94a7109f/uz2O2F_qbY3wefpY548qS.jpeg
Howe Tissue
user
Howe77
684b96403b733ba33368703c
Lu Wang
claimed_verified
2025-06-16T13:54:36.085Z
67ffb89774193e65a77cd914
/avatars/4302f06bcb1fa4c372b4d675459c4068.svg
Wang Lu
user
lwzzfzl
684b96403b733ba33368703d
Xipeng Qiu
Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture
  without Training
We introduce~Domain2Vec, a novel approach that decomposes any
dataset into a linear combination of several meta-domains, a new concept
designed to capture the key underlying features of datasets.
Domain2Vec maintains a vocabulary of meta-domains and uses a
classifier to decompose any given dataset into a domain vector that corresponds
to a distribution over this vocabulary. These domain vectors enable the
identification of the optimal data mixture for language model (LM) pretraining
in a training-free manner under the \textbf{Distribution
Alignment Assumption} (DA^{2}), which suggests that when
the data distributions of the training set and the validation set are better
aligned, a lower validation loss is achieved. Moreover, Domain2vec can
be seamlessly integrated into previous works to model the relationship between
domain vectors and LM performance, greatly enhancing the efficiency and
scalability of previous methods. Extensive experiments demonstrate that
Domain2Vec helps find the data mixture that enhances downstream task
performance with minimal computational overhead. Specifically,
Domain2Vec achieves the same validation loss on Pile-CC using only
51.5% of the computation required when training on the original mixture of
The Pile dataset. Under equivalent compute budget, Domain2Vec improves
downstream performance by an average of 2.83%.
Domain2Vec decomposes datasets into meta-domains to optimize language model pretraining and downstream performance with reduced computational cost.
Domain2Vec meta-domains domain vector distribution alignment assumption DA² language model pretraining downstream task performance Pile-CC The Pile dataset