DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of
  GPT-Generated Text
Large language models (LLMs) have notably enhanced the fluency and diversity
of machine-generated text. However, this progress also presents a significant
challenge in detecting the origin of a given text, and current research on
detection methods lags behind the rapid evolution of LLMs. Conventional
training-based methods have limitations in flexibility, particularly when
adapting to new domains, and they often lack explanatory power. To address this
gap, we propose a novel training-free detection strategy called Divergent
N-Gram Analysis (DNA-GPT). Given a text, we first truncate it in the middle and
then use only the preceding portion as input to the LLMs to regenerate the new
remaining parts. By analyzing the differences between the original and new
remaining parts through N-gram analysis in black-box or probability divergence
in white-box, we can clearly illustrate significant discrepancies between
machine-generated and human-written text. We conducted extensive experiments on
the most advanced LLMs from OpenAI, including text-davinci-003, GPT-3.5-turbo,
and GPT-4, as well as open-source models such as GPT-NeoX-20B and LLaMa-13B.
Results show that our zero-shot approach exhibits state-of-the-art performance
in distinguishing between human and GPT-generated text on four English and one
German dataset, outperforming OpenAI's own classifier, which is trained on
millions of text. Additionally, our methods provide reasonable explanations and
evidence to support our claim, which is a unique feature of explainable
detection. Our method is also robust under the revised text attack and can
additionally solve model sourcing. Codes are available at
https://github.com/Xianjun-Yang/DNA-GPT.
64755c0f7d131daf6340de73
Xianjun Yang
claimed_verified
2023-05-30T18:00:11.655Z
604137e5c15e823a685f2af2
https://cdn-avatars.huggingface.co/v1/production/uploads/604137e5c15e823a685f2af2/rrxZUyrbLpKqCmG6xtP8v.jpeg
Xianjun Yang
user
Xianjun
64755c0f7d131daf6340de74
Wei Cheng
claimed_verified
2023-05-30T14:15:24.520Z
6475f47d4766357252e7bc2a
/avatars/2fa3772318619258b25dd83533e74cea.svg
Wei Cheng
user
chengw07
64755c0f7d131daf6340de75
Linda Petzold
64755c0f7d131daf6340de76
William Yang Wang
claimed_verified
2023-05-31T07:24:35.872Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
64755c0f7d131daf6340de77
Haifeng Chen
DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of
  GPT-Generated Text
Large language models (LLMs) have notably enhanced the fluency and diversity
of machine-generated text. However, this progress also presents a significant
challenge in detecting the origin of a given text, and current research on
detection methods lags behind the rapid evolution of LLMs. Conventional
training-based methods have limitations in flexibility, particularly when
adapting to new domains, and they often lack explanatory power. To address this
gap, we propose a novel training-free detection strategy called Divergent
N-Gram Analysis (DNA-GPT). Given a text, we first truncate it in the middle and
then use only the preceding portion as input to the LLMs to regenerate the new
remaining parts. By analyzing the differences between the original and new
remaining parts through N-gram analysis in black-box or probability divergence
in white-box, we can clearly illustrate significant discrepancies between
machine-generated and human-written text. We conducted extensive experiments on
the most advanced LLMs from OpenAI, including text-davinci-003, GPT-3.5-turbo,
and GPT-4, as well as open-source models such as GPT-NeoX-20B and LLaMa-13B.
Results show that our zero-shot approach exhibits state-of-the-art performance
in distinguishing between human and GPT-generated text on four English and one
German dataset, outperforming OpenAI's own classifier, which is trained on
millions of text. Additionally, our methods provide reasonable explanations and
evidence to support our claim, which is a unique feature of explainable
detection. Our method is also robust under the revised text attack and can
additionally solve model sourcing. Codes are available at
https://github.com/Xianjun-Yang/DNA-GPT.
A novel training-free method, Divergent N-Gram Analysis (DNA-GPT), distinguishes machine-generated text from human-written text by analyzing N-gram divergence and outperforms existing methods in detecting text origins.
Divergent N-Gram Analysis DNA-GPT N-gram analysis probability divergence large language models LLMs text-davinci-003 GPT-3.5-turbo GPT-4 GPT-NeoX-20B LLaMa-13B zero-shot approach model sourcing explainable detection