Language Models are Injective and Hence Invertible
Transformer components such as non-linear activations and normalization are
inherently non-injective, suggesting that different inputs could map to the
same output and prevent exact recovery of the input from a model's
representations. In this paper, we challenge this view. First, we prove
mathematically that transformer language models mapping discrete input
sequences to their corresponding sequence of continuous representations are
injective and therefore lossless, a property established at initialization and
preserved during training. Second, we confirm this result empirically through
billions of collision tests on six state-of-the-art language models, and
observe no collisions. Third, we operationalize injectivity: we introduce
SipIt, the first algorithm that provably and efficiently reconstructs the exact
input text from hidden activations, establishing linear-time guarantees and
demonstrating exact invertibility in practice. Overall, our work establishes
injectivity as a fundamental and exploitable property of language models, with
direct implications for transparency, interpretability, and safe deployment.
68f5fa388589920bf4d3225c
Giorgos Nikolaou
claimed_verified
2025-10-23T15:01:13.658Z
66e56ad4c86016ecd972a9c5
/avatars/53ffcbe05c238d78ea227ed4e2b24b70.svg
Giorgos Nikolaou
user
giorgosnik02
68f5fa388589920bf4d3225d
Tommaso Mencattini
claimed_verified
2025-10-24T01:07:34.245Z
63ab16a6d7ee953f604ecd52
https://cdn-avatars.huggingface.co/v1/production/uploads/63ab16a6d7ee953f604ecd52/ujylOpczHKxU6Kfr-jGVr.png
Tommaso Mencattini
user
tmencatt
68f5fa388589920bf4d3225e
Donato Crisostomi
68f5fa388589920bf4d3225f
Andrea Santilli
claimed_verified
2025-10-23T15:01:19.541Z
5e8ef1f14957053f606489e6
https://cdn-avatars.huggingface.co/v1/production/uploads/1635502086699-5e8ef1f14957053f606489e6.jpeg
Andrea Santilli
user
teelinsan
68f5fa388589920bf4d32260
Yannis Panagakis
68f5fa388589920bf4d32261
Emanuele Rodola'
Language Models are Injective and Hence Invertible
Transformer components such as non-linear activations and normalization are
inherently non-injective, suggesting that different inputs could map to the
same output and prevent exact recovery of the input from a model's
representations. In this paper, we challenge this view. First, we prove
mathematically that transformer language models mapping discrete input
sequences to their corresponding sequence of continuous representations are
injective and therefore lossless, a property established at initialization and
preserved during training. Second, we confirm this result empirically through
billions of collision tests on six state-of-the-art language models, and
observe no collisions. Third, we operationalize injectivity: we introduce
SipIt, the first algorithm that provably and efficiently reconstructs the exact
input text from hidden activations, establishing linear-time guarantees and
demonstrating exact invertibility in practice. Overall, our work establishes
injectivity as a fundamental and exploitable property of language models, with
direct implications for transparency, interpretability, and safe deployment.
Transformer language models are proven to be injective, allowing exact input reconstruction from hidden activations, which has implications for transparency and safety.
transformer components non-linear activations normalization injective language models continuous representations collision tests SipIt linear-time guarantees exact invertibility