WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language
  Models
WalledEval is a comprehensive AI safety testing toolkit designed to evaluate
large language models (LLMs). It accommodates a diverse range of models,
including both open-weight and API-based ones, and features over 35 safety
benchmarks covering areas such as multilingual safety, exaggerated safety, and
prompt injections. The framework supports both LLM and judge benchmarking, and
incorporates custom mutators to test safety against various text-style
mutations such as future tense and paraphrasing. Additionally, WalledEval
introduces WalledGuard, a new, small and performant content moderation tool,
and SGXSTest, a benchmark for assessing exaggerated safety in cultural
contexts. We make WalledEval publicly available at
https://github.com/walledai/walledevalA.
66b489ffd84a01d665127b6e
Prannaya Gupta
admin_assigned
2024-08-08T10:34:42.340Z
60a546bdf9b53404e7806278
https://cdn-avatars.huggingface.co/v1/production/uploads/1621444268349-noauth.png
Prannaya Gupta
user
ThePyProgrammer
66b489ffd84a01d665127b6f
Le Qi Yau
claimed_verified
2024-08-09T07:48:18.061Z
64b35a5d4c3cc95a7553c360
/avatars/1d96d3fcf42c141d5fb9ef7bd691eb96.svg
Yau Le Qi
user
teraSurfer40141
66b489ffd84a01d665127b70
Hao Han Low
claimed_verified
2024-08-09T07:48:13.819Z
669745a9c1d9adc0c7daf1f5
/avatars/dac56ed2f6776074548b0132d097f13c.svg
Low Hao Han
user
redherring1
66b489ffd84a01d665127b71
I-Shiang Lee
claimed_verified
2024-08-09T07:48:16.221Z
668ff9e81e3f5b16cef10003
/avatars/4e10b275131058f588e228d09e5e7d24.svg
Lee I-Shiang
user
i-shiang
66b489ffd84a01d665127b72
Hugo Maximus Lim
66b489ffd84a01d665127b73
Yu Xin Teoh
66b489ffd84a01d665127b74
Jia Hng Koh
66b489ffd84a01d665127b75
Dar Win Liew
66b489ffd84a01d665127b76
Rishabh Bhardwaj
admin_assigned
2024-08-08T10:35:27.248Z
5f278507e923d665e616271b
https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
Rishabh Bhardwaj
user
RishabhBhardwaj
66b489ffd84a01d665127b77
Rajat Bhardwaj
admin_assigned
2024-08-08T10:35:34.653Z
6684cde7b6201f5294bdf41a
/avatars/3bc9e27a102f7a1a503b70b43e9e7034.svg
Rajat Bhardwaj
user
bhardwajrajat81
66b489ffd84a01d665127b78
Soujanya Poria
admin_assigned
2024-08-08T10:35:41.203Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/ZVSbhynzpQhVGq9kGywW6.png
Soujanya Poria
user
soujanyaporia
WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language
  Models
WalledEval is a comprehensive AI safety testing toolkit designed to evaluate
large language models (LLMs). It accommodates a diverse range of models,
including both open-weight and API-based ones, and features over 35 safety
benchmarks covering areas such as multilingual safety, exaggerated safety, and
prompt injections. The framework supports both LLM and judge benchmarking, and
incorporates custom mutators to test safety against various text-style
mutations such as future tense and paraphrasing. Additionally, WalledEval
introduces WalledGuard, a new, small and performant content moderation tool,
and SGXSTest, a benchmark for assessing exaggerated safety in cultural
contexts. We make WalledEval publicly available at
https://github.com/walledai/walledevalA.
WalledEval is a comprehensive safety testing toolkit for large language models using multiple safety benchmarks and custom mutators to assess performance in areas such as multilingual safety, exaggerated safety, and cultural contexts.
large language models safety benchmarks multilingual safety exaggerated safety prompt injections LLM judge benchmarking custom mutators WalledGuard SGXSTest