Extracting alignment data in open models
In this work, we show that it is possible to extract significant amounts of
alignment training data from a post-trained model -- useful to steer the model
to improve certain capabilities such as long-context reasoning, safety,
instruction following, and maths. While the majority of related work on
memorisation has focused on measuring success of training data extraction
through string matching, we argue that embedding models are better suited for
our specific goals. Distances measured through a high quality embedding model
can identify semantic similarities between strings that a different metric such
as edit distance will struggle to capture. In fact, in our investigation,
approximate string matching would have severely undercounted (by a conservative
estimate of 10times) the amount of data that can be extracted due to trivial
artifacts that deflate the metric. Interestingly, we find that models readily
regurgitate training data that was used in post-training phases such as SFT or
RL. We show that this data can be then used to train a base model, recovering a
meaningful amount of the original performance. We believe our work exposes a
possibly overlooked risk towards extracting alignment data. Finally, our work
opens up an interesting discussion on the downstream effects of distillation
practices: since models seem to be regurgitating aspects of their training set,
distillation can therefore be thought of as indirectly training on the model's
original dataset.
68f880b17669bcaeecce0e73
Federico Barbero
68f880b17669bcaeecce0e74
Xiangming Gu
68f880b17669bcaeecce0e75
Christopher A. Choquette-Choo
68f880b17669bcaeecce0e76
Chawin Sitawarin
68f880b17669bcaeecce0e77
Matthew Jagielski
68f880b17669bcaeecce0e78
Itay Yona
68f880b17669bcaeecce0e79
Petar Veličković
68f880b17669bcaeecce0e7a
Ilia Shumailov
68f880b17669bcaeecce0e7b
Jamie Hayes
Extracting alignment data in open models
In this work, we show that it is possible to extract significant amounts of
alignment training data from a post-trained model -- useful to steer the model
to improve certain capabilities such as long-context reasoning, safety,
instruction following, and maths. While the majority of related work on
memorisation has focused on measuring success of training data extraction
through string matching, we argue that embedding models are better suited for
our specific goals. Distances measured through a high quality embedding model
can identify semantic similarities between strings that a different metric such
as edit distance will struggle to capture. In fact, in our investigation,
approximate string matching would have severely undercounted (by a conservative
estimate of 10times) the amount of data that can be extracted due to trivial
artifacts that deflate the metric. Interestingly, we find that models readily
regurgitate training data that was used in post-training phases such as SFT or
RL. We show that this data can be then used to train a base model, recovering a
meaningful amount of the original performance. We believe our work exposes a
possibly overlooked risk towards extracting alignment data. Finally, our work
opens up an interesting discussion on the downstream effects of distillation
practices: since models seem to be regurgitating aspects of their training set,
distillation can therefore be thought of as indirectly training on the model's
original dataset.
Extracting alignment training data from post-trained models using embedding models reveals significant semantic similarities and potential risks in distillation practices.
embedding models long-context reasoning safety instruction following maths string matching edit distance semantic similarities SFT RL distillation