Evaluating Large Language Models for Detecting Antisemitism
Detecting hateful content is a challenging and important problem. Automated
tools, like machine-learning models, can help, but they require continuous
training to adapt to the ever-changing landscape of social media. In this work,
we evaluate eight open-source LLMs' capability to detect antisemitic content,
specifically leveraging in-context definition as a policy guideline. We explore
various prompting techniques and design a new CoT-like prompt, Guided-CoT.
Guided-CoT handles the in-context policy well, increasing performance across
all evaluated models, regardless of decoding configuration, model sizes, or
reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5.
Additionally, we examine LLM errors and introduce metrics to quantify semantic
divergence in model-generated rationales, revealing notable differences and
paradoxical behaviors among LLMs. Our experiments highlight the differences
observed across LLMs' utility, explainability, and reliability.
68d705200177a6054b0136ff
Jay Patel
68d705200177a6054b013700
Hrudayangam Mehta
68d705200177a6054b013701
Jeremy Blackburn
Evaluating Large Language Models for Detecting Antisemitism
Detecting hateful content is a challenging and important problem. Automated
tools, like machine-learning models, can help, but they require continuous
training to adapt to the ever-changing landscape of social media. In this work,
we evaluate eight open-source LLMs' capability to detect antisemitic content,
specifically leveraging in-context definition as a policy guideline. We explore
various prompting techniques and design a new CoT-like prompt, Guided-CoT.
Guided-CoT handles the in-context policy well, increasing performance across
all evaluated models, regardless of decoding configuration, model sizes, or
reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5.
Additionally, we examine LLM errors and introduce metrics to quantify semantic
divergence in model-generated rationales, revealing notable differences and
paradoxical behaviors among LLMs. Our experiments highlight the differences
observed across LLMs' utility, explainability, and reliability.
Evaluation of open-source LLMs for antisemitic content detection using in-context definition and a new Guided-CoT prompt shows improved performance and highlights differences in model utility, explainability, and reliability.
LLMs antisemitic content detection in-context definition prompting techniques Guided-CoT Llama 3.1 70B GPT-3.5 semantic divergence model-generated rationales utility explainability reliability