Detecting Data Contamination from Reinforcement Learning Post-training
  for Large Language Models
Data contamination poses a significant threat to the reliable evaluation of
Large Language Models (LLMs). This issue arises when benchmark samples may
inadvertently appear in training sets, compromising the validity of reported
performance. While detection methods have been developed for the pre-training
and Supervised Fine-Tuning stages, a critical research gap exists for the
increasingly significant phase of Reinforcement Learning (RL) post-training. As
RL post-training becomes pivotal for advancing LLM reasoning, the absence of
specialized contamination detection methods in this paradigm presents a
critical vulnerability. To address this, we conduct the first systematic study
of data detection within RL post-training scenario and propose Self-Critique.
Our method is motivated by a key observation: after RL phase, the output
entropy distribution of LLMs tends to collapse into highly specific and sparse
modes. Self-Critique probes for the underlying policy collapse, i.e., the
model's convergence to a narrow reasoning path, which causes this entropy
reduction. To facilitate this research, we also introduce RL-MIA, a benchmark
constructed to simulate this specific contamination scenario. Extensive
experiments show that Self-Critique significantly outperforms baseline methods
across multiple models and contamination tasks, achieving an AUC improvement of
up to 30%. Whereas existing methods are close to a random guess for RL-phase
contamination, our method makes detection possible.
68efba47d22df134b7f5c9d9
Yongding Tao
claimed_verified
2025-10-16T06:41:16.054Z
664b3aebfe822b08e62357f0
https://cdn-avatars.huggingface.co/v1/production/uploads/664b3aebfe822b08e62357f0/keEfUHXnkfwZ4o7588VdC.jpeg
Yongding Tao
user
YongdingTao
68efba47d22df134b7f5c9da
Tian Wang
68efba47d22df134b7f5c9db
Yihong Dong
68efba47d22df134b7f5c9dc
Huanyu Liu
68efba47d22df134b7f5c9dd
Kechi Zhang
68efba47d22df134b7f5c9de
Xiaolong Hu
68efba47d22df134b7f5c9df
Ge Li
Detecting Data Contamination from Reinforcement Learning Post-training
  for Large Language Models
Data contamination poses a significant threat to the reliable evaluation of
Large Language Models (LLMs). This issue arises when benchmark samples may
inadvertently appear in training sets, compromising the validity of reported
performance. While detection methods have been developed for the pre-training
and Supervised Fine-Tuning stages, a critical research gap exists for the
increasingly significant phase of Reinforcement Learning (RL) post-training. As
RL post-training becomes pivotal for advancing LLM reasoning, the absence of
specialized contamination detection methods in this paradigm presents a
critical vulnerability. To address this, we conduct the first systematic study
of data detection within RL post-training scenario and propose Self-Critique.
Our method is motivated by a key observation: after RL phase, the output
entropy distribution of LLMs tends to collapse into highly specific and sparse
modes. Self-Critique probes for the underlying policy collapse, i.e., the
model's convergence to a narrow reasoning path, which causes this entropy
reduction. To facilitate this research, we also introduce RL-MIA, a benchmark
constructed to simulate this specific contamination scenario. Extensive
experiments show that Self-Critique significantly outperforms baseline methods
across multiple models and contamination tasks, achieving an AUC improvement of
up to 30%. Whereas existing methods are close to a random guess for RL-phase
contamination, our method makes detection possible.
Self-Critique addresses data contamination in the RL post-training phase of LLMs by detecting policy collapse, outperforming existing methods with significant improvements in AUC.
Large Language Models Reinforcement Learning data contamination pre-training Supervised Fine-Tuning Self-Critique output entropy distribution policy collapse RL-MIA AUC