Hallucinating AI Hijacking Attack: Large Language Models and Malicious
  Code Recommenders
The research builds and evaluates the adversarial potential to introduce
copied code or hallucinated AI recommendations for malicious code in popular
code repositories. While foundational large language models (LLMs) from OpenAI,
Google, and Anthropic guard against both harmful behaviors and toxic strings,
previous work on math solutions that embed harmful prompts demonstrate that the
guardrails may differ between expert contexts. These loopholes would appear in
mixture of expert's models when the context of the question changes and may
offer fewer malicious training examples to filter toxic comments or recommended
offensive actions. The present work demonstrates that foundational models may
refuse to propose destructive actions correctly when prompted overtly but may
unfortunately drop their guard when presented with a sudden change of context,
like solving a computer programming challenge. We show empirical examples with
trojan-hosting repositories like GitHub, NPM, NuGet, and popular content
delivery networks (CDN) like jsDelivr which amplify the attack surface. In the
LLM's directives to be helpful, example recommendations propose application
programming interface (API) endpoints which a determined domain-squatter could
acquire and setup attack mobile infrastructure that triggers from the naively
copied code. We compare this attack to previous work on context-shifting and
contrast the attack surface as a novel version of "living off the land" attacks
in the malware literature. In the latter case, foundational language models can
hijack otherwise innocent user prompts to recommend actions that violate their
owners' safety policies when posed directly without the accompanying coding
support request.
6707460201f0e95af76fac44
David Noever
claimed_verified
2024-10-10T08:00:48.869Z
63136a82e29fb2e86d5e5bdd
https://cdn-avatars.huggingface.co/v1/production/uploads/63136a82e29fb2e86d5e5bdd/pFZDuQtzfUStovbwwZGvn.png
David Noever
user
dnoever
6707460201f0e95af76fac45
Forrest McKee
admin_assigned
2024-10-10T09:39:08.724Z
627bff703974b0ed6b2c340d
/avatars/12518ccef145f343548b3fd7be5efddc.svg
Forrest McKee
user
fgmckee
Hallucinating AI Hijacking Attack: Large Language Models and Malicious
  Code Recommenders
The research builds and evaluates the adversarial potential to introduce
copied code or hallucinated AI recommendations for malicious code in popular
code repositories. While foundational large language models (LLMs) from OpenAI,
Google, and Anthropic guard against both harmful behaviors and toxic strings,
previous work on math solutions that embed harmful prompts demonstrate that the
guardrails may differ between expert contexts. These loopholes would appear in
mixture of expert's models when the context of the question changes and may
offer fewer malicious training examples to filter toxic comments or recommended
offensive actions. The present work demonstrates that foundational models may
refuse to propose destructive actions correctly when prompted overtly but may
unfortunately drop their guard when presented with a sudden change of context,
like solving a computer programming challenge. We show empirical examples with
trojan-hosting repositories like GitHub, NPM, NuGet, and popular content
delivery networks (CDN) like jsDelivr which amplify the attack surface. In the
LLM's directives to be helpful, example recommendations propose application
programming interface (API) endpoints which a determined domain-squatter could
acquire and setup attack mobile infrastructure that triggers from the naively
copied code. We compare this attack to previous work on context-shifting and
contrast the attack surface as a novel version of "living off the land" attacks
in the malware literature. In the latter case, foundational language models can
hijack otherwise innocent user prompts to recommend actions that violate their
owners' safety policies when posed directly without the accompanying coding
support request.
adversarial potential copied code hallucinated AI recommendations malicious code large language models (LLMs) self repositories guardrails harmful prompts harmful behaviors toxic strings foundational models mixture adversarial potential copied code hallucinated AI recommendations malicious code large language models (LLMs) self repositories guardrails harmful prompts harmful behaviors toxic strings foundational models Trojan-hosting repositories GitHub NPM NuGet content delivery networks (CDN) jsDelivr API endpoints context-shifting "living off the land" attacks malware literature user prompts safety policies coding support request