Stealing Part of a Production Language Model
We introduce the first model-stealing attack that extracts precise,
nontrivial information from black-box production language models like OpenAI's
ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding
projection layer (up to symmetries) of a transformer model, given typical API
access. For under \20 USD, our attack extracts the entire projection matrix of
OpenAI's Ada and Babbage language models. We thereby confirm, for the first
time, that these black-box models have a hidden dimension of 1024 and 2048,
respectively. We also recover the exact hidden dimension size of the
gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to
recover the entire projection matrix. We conclude with potential defenses and
mitigations, and discuss the implications of possible future work that could
extend our attack.
65efc32c55f190f77e7b1d5b
Nicholas Carlini
admin_assigned
2024-03-12T12:07:28.322Z
6303fa9ba362e7e8b51d8f2a
/avatars/53e53c84f987989deb351dd2ae6ee558.svg
Nicholas Carlini
user
carlini
65efc32c55f190f77e7b1d5c
Daniel Paleka
admin_assigned
2024-03-12T12:07:34.833Z
62363277cd8e1f47c865190a
/avatars/fb42d42dedd4dde8df4109883cbe7ea9.svg
Daniel Paleka
user
dpaleka
65efc32c55f190f77e7b1d5d
Krishnamurthy Dj Dvijotham
65efc32c55f190f77e7b1d5e
Thomas Steinke
65efc32c55f190f77e7b1d5f
Jonathan Hayase
admin_assigned
2024-03-12T12:08:04.490Z
62f68be78465979f627b94b6
https://cdn-avatars.huggingface.co/v1/production/uploads/62f68be78465979f627b94b6/AfygEy8RaanUbWBkIxbey.jpeg
Jonathan Hayase
user
Jhayase
65efc32c55f190f77e7b1d60
A. Feder Cooper
admin_assigned
2024-03-12T12:08:10.892Z
659dba7a93fde4195d173334
/avatars/f7b7264a3007f7f0a85be6e3c698f432.svg
A. Feder Cooper
user
afcooper
65efc32c55f190f77e7b1d61
Katherine Lee
admin_assigned
2024-03-12T12:08:32.515Z
612a1e924f3c2ac50a06f950
/avatars/d424605e0d48327082114b878b91b161.svg
Katherine Lee
user
katherine1ee
65efc32c55f190f77e7b1d62
Matthew Jagielski
admin_assigned
2024-03-12T12:08:40.534Z
654a4a3b6a46bad8c0f2ab80
/avatars/08caf74525ea3e46fe58c20eb1f3d64d.svg
Matthew Jagielski
user
jagielski
65efc32c55f190f77e7b1d63
Milad Nasr
65efc32c55f190f77e7b1d64
Arthur Conmy
admin_assigned
2024-03-12T12:08:55.056Z
631ecc58daa9591e522e1494
/avatars/4f808fae966e808105e89712c97d90d2.svg
VConm
user
ArthurConmy
65efc32c55f190f77e7b1d65
Eric Wallace
admin_assigned
2024-03-12T12:09:47.872Z
63112d2431257261d20d5754
/avatars/502a68c0fb2f0c6989fe2869d0a7e3f4.svg
Eric Wallace
user
EricWallace
65efc32c55f190f77e7b1d66
David Rolnick
65efc32c55f190f77e7b1d67
Florian Tramèr
admin_assigned
2024-03-12T12:10:02.779Z
63568f18ba90b4ea9fe91cb5
/avatars/3e8b3c573e20cf80d329a312bfc34728.svg
Florian Tramer
user
ftramer
Stealing Part of a Production Language Model
We introduce the first model-stealing attack that extracts precise,
nontrivial information from black-box production language models like OpenAI's
ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding
projection layer (up to symmetries) of a transformer model, given typical API
access. For under \20 USD, our attack extracts the entire projection matrix of
OpenAI's Ada and Babbage language models. We thereby confirm, for the first
time, that these black-box models have a hidden dimension of 1024 and 2048,
respectively. We also recover the exact hidden dimension size of the
gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to
recover the entire projection matrix. We conclude with potential defenses and
mitigations, and discuss the implications of possible future work that could
extend our attack.
A model-stealing attack is introduced that can extract detailed information such as the embedding projection layer from black-box language models with minimal cost.
model-stealing attack embedding projection layer transformer model API access black-box models OpenAI's Ada OpenAI's Babbage gpt-3.5-turbo hidden dimension projection matrix potential defenses mitigations