People who frequently use ChatGPT for writing tasks are accurate and
  robust detectors of AI-generated text
In this paper, we study how well humans can detect text generated by
commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300
non-fiction English articles, label them as either human-written or
AI-generated, and provide paragraph-length explanations for their decisions.
Our experiments show that annotators who frequently use LLMs for writing tasks
excel at detecting AI-generated text, even without any specialized training or
feedback. In fact, the majority vote among five such "expert" annotators
misclassifies only 1 of 300 articles, significantly outperforming most
commercial and open-source detectors we evaluated even in the presence of
evasion tactics like paraphrasing and humanization. Qualitative analysis of the
experts' free-form explanations shows that while they rely heavily on specific
lexical clues ('AI vocabulary'), they also pick up on more complex phenomena
within the text (e.g., formality, originality, clarity) that are challenging to
assess for automatic detectors. We release our annotated dataset and code to
spur future research into both human and automated detection of AI-generated
text.
679a7a7ea3ffd2887d76a1e7
Jenna Russell
claimed_verified
2025-01-29T21:06:15.282Z
637519809ef4e3b87a5e88fb
/avatars/4465c9194c17f9b5e5a5a5e88d4a4656.svg
Jenna Russell
user
jjrussell10
679a7a7ea3ffd2887d76a1e8
Marzena Karpinska
extracted_confirmed
2025-01-31T03:42:57.813Z
62293b03acd5bef90e55c4ae
https://cdn-avatars.huggingface.co/v1/production/uploads/1667593978543-62293b03acd5bef90e55c4ae.png
Marzena Karpinska
user
marzena
679a7a7ea3ffd2887d76a1e9
Mohit Iyyer
People who frequently use ChatGPT for writing tasks are accurate and
  robust detectors of AI-generated text
In this paper, we study how well humans can detect text generated by
commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300
non-fiction English articles, label them as either human-written or
AI-generated, and provide paragraph-length explanations for their decisions.
Our experiments show that annotators who frequently use LLMs for writing tasks
excel at detecting AI-generated text, even without any specialized training or
feedback. In fact, the majority vote among five such "expert" annotators
misclassifies only 1 of 300 articles, significantly outperforming most
commercial and open-source detectors we evaluated even in the presence of
evasion tactics like paraphrasing and humanization. Qualitative analysis of the
experts' free-form explanations shows that while they rely heavily on specific
lexical clues ('AI vocabulary'), they also pick up on more complex phenomena
within the text (e.g., formality, originality, clarity) that are challenging to
assess for automatic detectors. We release our annotated dataset and code to
spur future research into both human and automated detection of AI-generated
text.
LLMs GPT-4 Claude human-written AI-generated paragraph-length explanations majority vote specialized training evasion tactics paraphrasing humanization lexical clues AI vocabulary formality originality clarity automatic detectors