Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Prompts
Evaluating jailbreak attacks is challenging when prompts are not overtly
harmful or fail to induce harmful outputs. Unfortunately, many existing
red-teaming datasets contain such unsuitable prompts. To evaluate attacks
accurately, these datasets need to be assessed and cleaned for maliciousness.
However, existing malicious content detection methods rely on either manual
annotation, which is labor-intensive, or large language models (LLMs), which
have inconsistent accuracy in harmful types. To balance accuracy and
efficiency, we propose a hybrid evaluation framework named MDH (Malicious
content Detection based on LLMs with Human assistance) that combines LLM-based
annotation with minimal human oversight, and apply it to dataset cleaning and
detection of jailbroken responses. Furthermore, we find that well-crafted
developer messages can significantly boost jailbreak success, leading us to
propose two new strategies: D-Attack, which leverages context simulation, and
DH-CoT, which incorporates hijacked chains of thought. The Codes, datasets,
judgements, and detection results will be released in github repository:
https://github.com/AlienZhang1996/DH-CoT.
689fe738a4caabb4320e6067
Chiyu Zhang
claimed_verified
2025-08-18T06:55:29.169Z
637f6b5752229c6392138545
https://cdn-avatars.huggingface.co/v1/production/uploads/1669295079706-637f6b5752229c6392138545.jpeg
Zhang
user
AlienZhang1996
689fe738a4caabb4320e6068
Lu Zhou
689fe738a4caabb4320e6069
Xiaogang Xu
689fe738a4caabb4320e606a
Jiafei Wu
689fe738a4caabb4320e606b
Liming Fang
689fe738a4caabb4320e606c
Zhe Liu
Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Prompts
Evaluating jailbreak attacks is challenging when prompts are not overtly
harmful or fail to induce harmful outputs. Unfortunately, many existing
red-teaming datasets contain such unsuitable prompts. To evaluate attacks
accurately, these datasets need to be assessed and cleaned for maliciousness.
However, existing malicious content detection methods rely on either manual
annotation, which is labor-intensive, or large language models (LLMs), which
have inconsistent accuracy in harmful types. To balance accuracy and
efficiency, we propose a hybrid evaluation framework named MDH (Malicious
content Detection based on LLMs with Human assistance) that combines LLM-based
annotation with minimal human oversight, and apply it to dataset cleaning and
detection of jailbroken responses. Furthermore, we find that well-crafted
developer messages can significantly boost jailbreak success, leading us to
propose two new strategies: D-Attack, which leverages context simulation, and
DH-CoT, which incorporates hijacked chains of thought. The Codes, datasets,
judgements, and detection results will be released in github repository:
https://github.com/AlienZhang1996/DH-CoT.
A hybrid framework combining LLMs and human oversight is proposed to clean datasets and detect jailbreak attacks, with new strategies to enhance jailbreak success.
red-teaming datasets large language models LLMs hybrid evaluation framework MDH malicious content detection dataset cleaning jailbroken responses D-Attack context simulation DH-CoT hijacked chains of thought