Data Mixture Inference: What do BPE Tokenizers Reveal about their
  Training Data?
The pretraining data of today's strongest language models is opaque. In
particular, little is known about the proportions of various domains or
languages represented. In this work, we tackle a task which we call data
mixture inference, which aims to uncover the distributional make-up of training
data. We introduce a novel attack based on a previously overlooked source of
information -- byte-pair encoding (BPE) tokenizers, used by the vast majority
of modern language models. Our key insight is that the ordered list of merge
rules learned by a BPE tokenizer naturally reveals information about the token
frequencies in its training data: the first merge is the most common byte pair,
the second is the most common pair after merging the first token, and so on.
Given a tokenizer's merge list along with data samples for each category of
interest, we formulate a linear program that solves for the proportion of each
category in the tokenizer's training set. Importantly, to the extent to which
tokenizer training data is representative of the pretraining data, we
indirectly learn about the pretraining data. In controlled experiments, we show
that our attack recovers mixture ratios with high precision for tokenizers
trained on known mixtures of natural languages, programming languages, and data
sources. We then apply our approach to off-the-shelf tokenizers released with
recent LMs. We confirm much publicly disclosed information about these models,
and also make several new inferences: GPT-4o's tokenizer is much more
multilingual than its predecessors, training on 39% non-English data; Llama3
extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT-3.5's and
Claude's tokenizers are trained on predominantly code (~60%). We hope our work
sheds light on current design practices for pretraining data, and inspires
continued research into data mixture inference for LMs.
66a0f22917555c302003b936
Jonathan Hayase
admin_assigned
2024-07-26T12:18:19.787Z
62f68be78465979f627b94b6
https://cdn-avatars.huggingface.co/v1/production/uploads/62f68be78465979f627b94b6/AfygEy8RaanUbWBkIxbey.jpeg
Jonathan Hayase
user
Jhayase
66a0f22917555c302003b937
Alisa Liu
admin_assigned
2024-07-26T12:18:55.764Z
62449da46b55ef458d430371
/avatars/b95954a88c7b15524a3d7776a4abe083.svg
Alisa Liu
user
alisawuffles
66a0f22917555c302003b938
Yejin Choi
admin_assigned
2024-07-26T12:19:14.024Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
66a0f22917555c302003b939
Sewoong Oh
66a0f22917555c302003b93a
Noah A. Smith
Data Mixture Inference: What do BPE Tokenizers Reveal about their
  Training Data?
The pretraining data of today's strongest language models is opaque. In
particular, little is known about the proportions of various domains or
languages represented. In this work, we tackle a task which we call data
mixture inference, which aims to uncover the distributional make-up of training
data. We introduce a novel attack based on a previously overlooked source of
information -- byte-pair encoding (BPE) tokenizers, used by the vast majority
of modern language models. Our key insight is that the ordered list of merge
rules learned by a BPE tokenizer naturally reveals information about the token
frequencies in its training data: the first merge is the most common byte pair,
the second is the most common pair after merging the first token, and so on.
Given a tokenizer's merge list along with data samples for each category of
interest, we formulate a linear program that solves for the proportion of each
category in the tokenizer's training set. Importantly, to the extent to which
tokenizer training data is representative of the pretraining data, we
indirectly learn about the pretraining data. In controlled experiments, we show
that our attack recovers mixture ratios with high precision for tokenizers
trained on known mixtures of natural languages, programming languages, and data
sources. We then apply our approach to off-the-shelf tokenizers released with
recent LMs. We confirm much publicly disclosed information about these models,
and also make several new inferences: GPT-4o's tokenizer is much more
multilingual than its predecessors, training on 39% non-English data; Llama3
extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT-3.5's and
Claude's tokenizers are trained on predominantly code (~60%). We hope our work
sheds light on current design practices for pretraining data, and inspires
continued research into data mixture inference for LMs.
Data mixture inference using byte-pair encoding (BPE) tokenizers uncovers the distribution of training data in language models, revealing proportions of various domains and languages.
byte-pair encoding BPE tokenizers token frequencies merge rules linear program natural languages programming languages data sources multilingual code