Efficient Detection of Toxic Prompts in Large Language Models
Large language models (LLMs) like ChatGPT and Gemini have significantly
advanced natural language processing, enabling various applications such as
chatbots and automated content generation. However, these models can be
exploited by malicious individuals who craft toxic prompts to elicit harmful or
unethical responses. These individuals often employ jailbreaking techniques to
bypass safety mechanisms, highlighting the need for robust toxic prompt
detection methods. Existing detection techniques, both blackbox and whitebox,
face challenges related to the diversity of toxic prompts, scalability, and
computational efficiency. In response, we propose ToxicDetector, a lightweight
greybox method designed to efficiently detect toxic prompts in LLMs.
ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding
vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP)
classifier for prompt classification. Our evaluation on various versions of the
LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector
achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%,
outperforming state-of-the-art methods. Additionally, ToxicDetector's
processing time of 0.0780 seconds per prompt makes it highly suitable for
real-time applications. ToxicDetector achieves high accuracy, efficiency, and
scalability, making it a practical method for toxic prompt detection in LLMs.
66c6b1d759ff3e4811a05026
Yi Liu
claimed_verified
2024-08-22T19:50:28.173Z
63c6b13950cc81901da4da6c
/avatars/23f2016e7c7ad902c5750e2ddc0b6358.svg
YI LIU
user
sumleo
66c6b1d759ff3e4811a05027
Junzhe Yu
admin_assigned
2024-08-27T13:31:40.415Z
65e6c52354d7e3a708444bcc
/avatars/b1f6dc0e3c4b4b103e60aa306c8b7c11.svg
Junzhe Yu
user
kirkyDig
66c6b1d759ff3e4811a05028
Huijia Sun
66c6b1d759ff3e4811a05029
Ling Shi
66c6b1d759ff3e4811a0502a
Gelei Deng
admin_assigned
2024-08-27T13:32:12.147Z
64ae49347a9df1bd416f7e21
/avatars/f8226d99e8f272906e8934c3487ca6e2.svg
Deng
user
Gelei
66c6b1d759ff3e4811a0502b
Yuqi Chen
admin_assigned
2024-08-27T13:32:18.190Z
63914edd2e1b430e96cf4d59
https://cdn-avatars.huggingface.co/v1/production/uploads/1670467355858-63914edd2e1b430e96cf4d59.jpeg
Yuqi Chen
user
YuqiChen
66c6b1d759ff3e4811a0502c
Yang Liu
Efficient Detection of Toxic Prompts in Large Language Models
Large language models (LLMs) like ChatGPT and Gemini have significantly
advanced natural language processing, enabling various applications such as
chatbots and automated content generation. However, these models can be
exploited by malicious individuals who craft toxic prompts to elicit harmful or
unethical responses. These individuals often employ jailbreaking techniques to
bypass safety mechanisms, highlighting the need for robust toxic prompt
detection methods. Existing detection techniques, both blackbox and whitebox,
face challenges related to the diversity of toxic prompts, scalability, and
computational efficiency. In response, we propose ToxicDetector, a lightweight
greybox method designed to efficiently detect toxic prompts in LLMs.
ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding
vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP)
classifier for prompt classification. Our evaluation on various versions of the
LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector
achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%,
outperforming state-of-the-art methods. Additionally, ToxicDetector's
processing time of 0.0780 seconds per prompt makes it highly suitable for
real-time applications. ToxicDetector achieves high accuracy, efficiency, and
scalability, making it a practical method for toxic prompt detection in LLMs.
ToxicDetector is a lightweight greybox method using LLMs, embeddings, and MLP classifiers to efficiently detect toxic prompts with high accuracy and low false positives.
Multilayer Perceptron (MLP) embedding vectors prompt classification ToxicDetector Large Language Models (LLMs) toxic concept prompts