DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios
Yao Huang, Yitong Sun, Yichi Zhang, Ruochen Zhang, Yinpeng Dong, Xingxing Wei
Large Language Models (LLMs) have achieved remarkable proficiency across a wide range of tasks, yet with these advances come emergent behaviors, such as sophisticated deception, where models may subtly mislead or manipulate responses, posing significant risks to their trustworthiness. While LLM performance has been extensively studied, systematic evaluations of deceptive tendencies across diverse real-world contexts remain limited. To address this gap, we introduce a comprehensive evaluation framework spanning five critical domains: economy, healthcare, education, social interaction, and entertainment, selected for their profound societal relevance. Our benchmark encompasses 150 scenarios, supported by a dataset of 1.5k samples, ensuring a thorough and holistic analysis. Additionally, we investigate model behavior under varied feedback mechanisms: no feedback, positive reinforcement (reward), negative feedback (pressure), and multi-turn reinforced positive or negative feedback loops, to elucidate how these conditions influence deceptive tendencies. Through extensive experiments, we broadly evaluate a diverse set of models, including standard LLMs and reasoning LLMs, revealing critical vulnerabilities, such as amplified deceptive responses under specific feedback dynamics, highlighting the urgent need for advanced mitigation strategies.