Ruby Teaming: Improving Quality Diversity Search with Memory for
  Automated Red Teaming
We propose Ruby Teaming, a method that improves on Rainbow Teaming by
including a memory cache as its third dimension. The memory dimension provides
cues to the mutator to yield better-quality prompts, both in terms of attack
success rate (ASR) and quality diversity. The prompt archive generated by Ruby
Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of
quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on
Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI),
respectively.
667987754e7665bb49695f52
Vernon Toh Yan Han
667987754e7665bb49695f53
Rishabh Bhardwaj
claimed_verified
2024-09-19T07:08:26.864Z
5f278507e923d665e616271b
https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
Rishabh Bhardwaj
user
RishabhBhardwaj
667987754e7665bb49695f54
Soujanya Poria
claimed_verified
2024-08-22T19:52:34.994Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/ZVSbhynzpQhVGq9kGywW6.png
Soujanya Poria
user
soujanyaporia
Ruby Teaming: Improving Quality Diversity Search with Memory for
  Automated Red Teaming
We propose Ruby Teaming, a method that improves on Rainbow Teaming by
including a memory cache as its third dimension. The memory dimension provides
cues to the mutator to yield better-quality prompts, both in terms of attack
success rate (ASR) and quality diversity. The prompt archive generated by Ruby
Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of
quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on
Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI),
respectively.
Ruby Teaming introduces a memory cache to improve prompt quality and attack success rate compared to Rainbow Teaming.
memory cache prompt archive attack success rate quality diversity Shannon's Evenness Index Simpson's Diversity Index