Machine Text Detectors are Membership Inference Attacks
Although membership inference attacks (MIAs) and machine-generated text
detection target different goals, identifying training samples and synthetic
texts, their methods often exploit similar signals based on a language model's
probability distribution. Despite this shared methodological foundation, the
two tasks have been independently studied, which may lead to conclusions that
overlook stronger methods and valuable insights developed in the other task. In
this work, we theoretically and empirically investigate the transferability,
i.e., how well a method originally developed for one task performs on the
other, between MIAs and machine text detection. For our theoretical
contribution, we prove that the metric that achieves the asymptotically highest
performance on both tasks is the same. We unify a large proportion of the
existing literature in the context of this optimal metric and hypothesize that
the accuracy with which a given method approximates this metric is directly
correlated with its transferability. Our large-scale empirical experiments,
including 7 state-of-the-art MIA methods and 5 state-of-the-art machine text
detectors across 13 domains and 10 generators, demonstrate very strong rank
correlation (rho > 0.6) in cross-task performance. We notably find that
Binoculars, originally designed for machine text detection, achieves
state-of-the-art performance on MIA benchmarks as well, demonstrating the
practical impact of the transferability. Our findings highlight the need for
greater cross-task awareness and collaboration between the two research
communities. To facilitate cross-task developments and fair evaluations, we
introduce MINT, a unified evaluation suite for MIAs and machine-generated text
detection, with implementation of 15 recent methods from both tasks.
68f9ab5bb9b2e4ae046738bb
Ryuto Koike
68f9ab5bb9b2e4ae046738bc
Liam Dugan
claimed_verified
2025-10-23T14:47:10.561Z
6509e1e01b3694179dee256e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sUbok2yloLElaebkps3lT.png
Liam Dugan
user
liamdugan
68f9ab5bb9b2e4ae046738bd
Masahiro Kaneko
68f9ab5bb9b2e4ae046738be
Chris Callison-Burch
68f9ab5bb9b2e4ae046738bf
Naoaki Okazaki
Machine Text Detectors are Membership Inference Attacks
Although membership inference attacks (MIAs) and machine-generated text
detection target different goals, identifying training samples and synthetic
texts, their methods often exploit similar signals based on a language model's
probability distribution. Despite this shared methodological foundation, the
two tasks have been independently studied, which may lead to conclusions that
overlook stronger methods and valuable insights developed in the other task. In
this work, we theoretically and empirically investigate the transferability,
i.e., how well a method originally developed for one task performs on the
other, between MIAs and machine text detection. For our theoretical
contribution, we prove that the metric that achieves the asymptotically highest
performance on both tasks is the same. We unify a large proportion of the
existing literature in the context of this optimal metric and hypothesize that
the accuracy with which a given method approximates this metric is directly
correlated with its transferability. Our large-scale empirical experiments,
including 7 state-of-the-art MIA methods and 5 state-of-the-art machine text
detectors across 13 domains and 10 generators, demonstrate very strong rank
correlation (rho > 0.6) in cross-task performance. We notably find that
Binoculars, originally designed for machine text detection, achieves
state-of-the-art performance on MIA benchmarks as well, demonstrating the
practical impact of the transferability. Our findings highlight the need for
greater cross-task awareness and collaboration between the two research
communities. To facilitate cross-task developments and fair evaluations, we
introduce MINT, a unified evaluation suite for MIAs and machine-generated text
detection, with implementation of 15 recent methods from both tasks.
Theoretical and empirical investigation shows strong transferability between membership inference attacks and machine-generated text detection, highlighting the need for cross-task collaboration and introducing MINT for unified evaluation.
membership inference attacks machine-generated text detection language model probability distribution transferability Binoculars MINT