Evaluating Large Language Models for Detecting Antisemitism
Detecting hateful content is a challenging and important problem. Automated
tools, like machine-learning models, can help, but they require continuous
training to adapt to the ever-changing landscape of social media. In this work,
we evaluate eight open-source LLMs' capability to detect antisemitic content,
specifically leveraging in-context definition as a policy guideline. We explore
various prompting techniques and design a new CoT-like prompt, Guided-CoT.
Guided-CoT handles the in-context policy well, increasing performance across
all evaluated models, regardless of decoding configuration, model sizes, or
reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5.
Additionally, we examine LLM errors and introduce metrics to quantify semantic
divergence in model-generated rationales, revealing notable differences and
paradoxical behaviors among LLMs. Our experiments highlight the differences
observed across LLMs' utility, explainability, and reliability.
68d705200177a6054b0136ff
Jay Patel
claimed_verified
2025-09-29T06:01:31.087Z
64b965493269cb9386c507c3
https://cdn-avatars.huggingface.co/v1/production/uploads/64b965493269cb9386c507c3/v_HEnaVVg9OzMZGDKfAk0.jpeg
Jay Patel
user
jpatel0057
68d705200177a6054b013700
Hrudayangam Mehta
68d705200177a6054b013701
Jeremy Blackburn
Evaluating Large Language Models for Detecting Antisemitism
Detecting hateful content is a challenging and important problem. Automated
tools, like machine-learning models, can help, but they require continuous
training to adapt to the ever-changing landscape of social media. In this work,
we evaluate eight open-source LLMs' capability to detect antisemitic content,
specifically leveraging in-context definition as a policy guideline. We explore
various prompting techniques and design a new CoT-like prompt, Guided-CoT.
Guided-CoT handles the in-context policy well, increasing performance across
all evaluated models, regardless of decoding configuration, model sizes, or
reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5.
Additionally, we examine LLM errors and introduce metrics to quantify semantic
divergence in model-generated rationales, revealing notable differences and
paradoxical behaviors among LLMs. Our experiments highlight the differences
observed across LLMs' utility, explainability, and reliability.
Evaluation of open-source LLMs for antisemitic content detection using in-context definition and a new Guided-CoT prompt shows improved performance and highlights differences in model utility, explainability, and reliability.
LLMs antisemitic content detection in-context definition prompting techniques Guided-CoT Llama 3.1 70B GPT-3.5 semantic divergence model-generated rationales utility explainability reliability