Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models
Large language models (LLMs) have demonstrated impressive capabilities across
various natural language processing (NLP) tasks in recent years. However, their
susceptibility to jailbreaks and perturbations necessitates additional
evaluations. Many LLMs are multilingual, but safety-related training data
contains mainly high-resource languages like English. This can leave them
vulnerable to perturbations in low-resource languages such as Polish. We show
how surprisingly strong attacks can be cheaply created by altering just a few
characters and using a small proxy model for word importance calculation. We
find that these character and word-level attacks drastically alter the
predictions of different LLMs, suggesting a potential vulnerability that can be
used to circumvent their internal safety mechanisms. We validate our attack
construction methodology on Polish, a low-resource language, and find potential
vulnerabilities of LLMs in this language. Additionally, we show how it can be
extended to other languages. We release the created datasets and code for
further research.
6847ea583ec10bdd8ab4e05a
Maciej Chrabąszcz
claimed_verified
2025-06-10T08:42:31.860Z
635270e36cfb8f14981312e7
https://cdn-avatars.huggingface.co/v1/production/uploads/635270e36cfb8f14981312e7/FpqjdhABEDuxWR8k2zOwA.jpeg
Maciej Chrabąszcz
user
mchraba
6847ea583ec10bdd8ab4e05b
Katarzyna Lorenc
extracted_pending
2025-06-10T08:18:33.059Z
66dab47f8506f9b6cf5f08ed
/avatars/e6ba87adbaacdeccf8c4818596c655d0.svg
LLM Attack
user
llmAttack
6847ea583ec10bdd8ab4e05c
Karolina Seweryn
claimed_verified
2025-06-17T15:13:14.513Z
63e3b177737f1e4237505204
https://cdn-avatars.huggingface.co/v1/production/uploads/63e3b177737f1e4237505204/KTRd9_zkavGgi1J7AFuKc.jpeg
Karolina Seweryn
user
karolinaseweryn
Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models
Large language models (LLMs) have demonstrated impressive capabilities across
various natural language processing (NLP) tasks in recent years. However, their
susceptibility to jailbreaks and perturbations necessitates additional
evaluations. Many LLMs are multilingual, but safety-related training data
contains mainly high-resource languages like English. This can leave them
vulnerable to perturbations in low-resource languages such as Polish. We show
how surprisingly strong attacks can be cheaply created by altering just a few
characters and using a small proxy model for word importance calculation. We
find that these character and word-level attacks drastically alter the
predictions of different LLMs, suggesting a potential vulnerability that can be
used to circumvent their internal safety mechanisms. We validate our attack
construction methodology on Polish, a low-resource language, and find potential
vulnerabilities of LLMs in this language. Additionally, we show how it can be
extended to other languages. We release the created datasets and code for
further research.
Character and word-level attacks using a proxy model reveal vulnerabilities in LLMs across languages, particularly in low-resource languages like Polish.
large language models natural language processing jailbreaks perturbations multilingual safety-related training data high-resource languages low-resource languages character-level attacks word-level attacks word importance calculation internal safety mechanisms