Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability
  in Knowledge and Safety with DuET-PD
Large Language Models (LLMs) can struggle to balance gullibility to
misinformation and resistance to valid corrections in persuasive dialogues, a
critical challenge for reliable deployment. We introduce DuET-PD (Dual
Evaluation for Trust in Persuasive Dialogues), a framework evaluating
multi-turn stance-change dynamics across dual dimensions: persuasion type
(corrective/misleading) and domain (knowledge via MMLU-Pro, and safety via
SALAD-Bench). We find that even a state-of-the-art model like GPT-4o achieves
only 27.32% accuracy in MMLU-Pro under sustained misleading persuasions.
Moreover, results reveal a concerning trend of increasing sycophancy in newer
open-source models. To address this, we introduce Holistic DPO, a training
approach balancing positive and negative persuasion examples. Unlike prompting
or resist-only training, Holistic DPO enhances both robustness to
misinformation and receptiveness to corrections, improving
Llama-3.1-8B-Instruct's accuracy under misleading persuasion in safety contexts
from 4.21% to 76.54%. These contributions offer a pathway to developing more
reliable and adaptable LLMs for multi-turn dialogue. Code is available at
https://github.com/Social-AI-Studio/DuET-PD.
68b102c7b19c5400014848c4
Bryan Chen Zhengyu Tan
claimed_verified
2025-08-29T08:02:39.201Z
6312090b3a773344e06b8820
/avatars/d6706a2bac22b8bd8defda33488bd5b7.svg
Bryan Tan (Chen Zhengyu)
user
Incomple
68b102c7b19c5400014848c5
Daniel Wai Kit Chin
68b102c7b19c5400014848c6
Zhengyuan Liu
68b102c7b19c5400014848c7
Nancy F. Chen
68b102c7b19c5400014848c8
Roy Ka-Wei Lee
Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability
  in Knowledge and Safety with DuET-PD
Large Language Models (LLMs) can struggle to balance gullibility to
misinformation and resistance to valid corrections in persuasive dialogues, a
critical challenge for reliable deployment. We introduce DuET-PD (Dual
Evaluation for Trust in Persuasive Dialogues), a framework evaluating
multi-turn stance-change dynamics across dual dimensions: persuasion type
(corrective/misleading) and domain (knowledge via MMLU-Pro, and safety via
SALAD-Bench). We find that even a state-of-the-art model like GPT-4o achieves
only 27.32% accuracy in MMLU-Pro under sustained misleading persuasions.
Moreover, results reveal a concerning trend of increasing sycophancy in newer
open-source models. To address this, we introduce Holistic DPO, a training
approach balancing positive and negative persuasion examples. Unlike prompting
or resist-only training, Holistic DPO enhances both robustness to
misinformation and receptiveness to corrections, improving
Llama-3.1-8B-Instruct's accuracy under misleading persuasion in safety contexts
from 4.21% to 76.54%. These contributions offer a pathway to developing more
reliable and adaptable LLMs for multi-turn dialogue. Code is available at
https://github.com/Social-AI-Studio/DuET-PD.
DuET-PD evaluates LLMs in persuasive dialogues, revealing challenges with misinformation and corrections, and introduces Holistic DPO to improve model reliability.
Large Language Models DuET-PD stance-change dynamics persuasion type domain MMLU-Pro SALAD-Bench GPT-4o sycophancy Holistic DPO robustness receptiveness Llama-3.1-8B-Instruct