Measuring memorization in RLHF for code completion
Reinforcement learning with human feedback (RLHF) has become the dominant
method to align large models to user preferences. Unlike fine-tuning, for which
there are many studies regarding training data memorization, it is not clear
how memorization is affected by or introduced in the RLHF alignment process.
Understanding this relationship is important as real user data may be collected
and used to align large models; if user data is memorized during RLHF and later
regurgitated, this could raise privacy concerns. In this work, we analyze how
training data memorization can surface and propagate through each phase of
RLHF. We focus our study on code completion models, as code completion is one
of the most popular use cases for large language models. We find that RLHF
significantly decreases the chance that data used for reward modeling and
reinforcement learning is memorized, in comparison to aligning via directly
fine-tuning on this data, but that examples already memorized during the
fine-tuning stage of RLHF, will, in the majority of cases, remain memorized
after RLHF.
6674aa5266539bc0a0e09363
Aneesh Pappu
6674aa5266539bc0a0e09364
Billy Porter
extracted_confirmed
2024-06-20T22:20:03.982Z
6674a6bea845e4470f065409
/avatars/d353fbed5294b017c964a726177a4f06.svg
Billy Porter
user
billyporter
6674aa5266539bc0a0e09365
Ilia Shumailov
6674aa5266539bc0a0e09366
Jamie Hayes
Measuring memorization in RLHF for code completion
Reinforcement learning with human feedback (RLHF) has become the dominant
method to align large models to user preferences. Unlike fine-tuning, for which
there are many studies regarding training data memorization, it is not clear
how memorization is affected by or introduced in the RLHF alignment process.
Understanding this relationship is important as real user data may be collected
and used to align large models; if user data is memorized during RLHF and later
regurgitated, this could raise privacy concerns. In this work, we analyze how
training data memorization can surface and propagate through each phase of
RLHF. We focus our study on code completion models, as code completion is one
of the most popular use cases for large language models. We find that RLHF
significantly decreases the chance that data used for reward modeling and
reinforcement learning is memorized, in comparison to aligning via directly
fine-tuning on this data, but that examples already memorized during the
fine-tuning stage of RLHF, will, in the majority of cases, remain memorized
after RLHF.
Reinforcement Learning with Human Feedback decreases the chance of data memorization compared to direct fine-tuning, though previously memorized examples may persist.
Reinforcement learning with human feedback RLHF reward modeling memorization code completion models large language models fine-tuning