Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient
  LLMs Under Compression
Compressing high-capability Large Language Models (LLMs) has emerged as a
favored strategy for resource-efficient inferences. While state-of-the-art
(SoTA) compression methods boast impressive advancements in preserving benign
task performance, the potential risks of compression in terms of safety and
trustworthiness have been largely neglected. This study conducts the first,
thorough evaluation of three (3) leading LLMs using five (5) SoTA compression
techniques across eight (8) trustworthiness dimensions. Our experiments
highlight the intricate interplay between compression and trustworthiness,
revealing some interesting patterns. We find that quantization is currently a
more effective approach than pruning in achieving efficiency and
trustworthiness simultaneously. For instance, a 4-bit quantized model retains
the trustworthiness of its original counterpart, but model pruning
significantly degrades trustworthiness, even at 50% sparsity. Moreover,
employing quantization within a moderate bit range could unexpectedly improve
certain trustworthiness dimensions such as ethics and fairness. Conversely,
extreme quantization to very low bit levels (3 bits) tends to significantly
reduce trustworthiness. This increased risk cannot be uncovered by looking at
benign performance alone, in turn, mandating comprehensive trustworthiness
evaluation in practice. These findings culminate in practical recommendations
for simultaneously achieving high utility, efficiency, and trustworthiness in
LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
66024c5f73c7ce480391b863
Junyuan Hong
claimed_verified
2024-03-26T07:54:17.032Z
6400cf982b67d27affce2d89
https://cdn-avatars.huggingface.co/v1/production/uploads/6400cf982b67d27affce2d89/Vs042d2M-iV2wk9Q_Jqh3.jpeg
Junyuan Hong
user
jyhong836
66024c5f73c7ce480391b864
Jinhao Duan
claimed_verified
2024-03-26T13:58:51.787Z
6423855c2d5965e7229ebe57
/avatars/babb01f5f2329cc95e13e33b1ec5cf25.svg
jinhaoduan
user
jhao
66024c5f73c7ce480391b865
Chenhui Zhang
claimed_verified
2024-03-26T07:54:19.015Z
64522233ea94bf023430dd95
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CVDqDeJ_fLTULhCTTSogb.png
Chenhui Zhang
user
danielz01
66024c5f73c7ce480391b866
Zhangheng Li
66024c5f73c7ce480391b867
Chulin Xie
admin_assigned
2024-03-26T09:12:24.428Z
645867e3ead43697df2ca5fb
/avatars/a0f0e5d975898413be456667c0cbd2a8.svg
Chulin Xie
user
alphapav
66024c5f73c7ce480391b868
Kelsey Lieberman
admin_assigned
2024-03-26T09:12:30.611Z
633eeebf0ffccea68bd7c4c7
/avatars/4959ad22f0f4eaf5712ba7a38f74679a.svg
Kelsey Lieberman
user
klieberman
66024c5f73c7ce480391b869
James Diffenderfer
66024c5f73c7ce480391b86a
Brian Bartoldson
admin_assigned
2024-03-26T09:13:19.748Z
641a7ecb91e3376a057adf89
/avatars/f407adae3e9df964a469cfb350115eb8.svg
brian bartoldson
user
bartoldson
66024c5f73c7ce480391b86b
Ajay Jaiswal
admin_assigned
2024-03-26T09:13:26.229Z
64ad7afdafb6aa5534221c96
/avatars/c1c83d8f5ca683ce60fd4d50c028471c.svg
Ajay Jaiswal
user
Ajay1994
66024c5f73c7ce480391b86c
Kaidi Xu
admin_assigned
2024-03-26T09:14:11.603Z
65aaa5b4d2adc31ee3eab350
https://cdn-avatars.huggingface.co/v1/production/uploads/65aaa5b4d2adc31ee3eab350/vzDrR7JVCEShn_Vk-NCgT.jpeg
Kaidi Xu
user
KaidiXu1
66024c5f73c7ce480391b86d
Bhavya Kailkhura
admin_assigned
2024-03-26T09:19:33.378Z
65cb79db6427380bc21261e2
/avatars/a003eb5d0955417329c1a4170ae65879.svg
Bhavya Kailkhura
user
bhavyakailkhura
66024c5f73c7ce480391b86e
Dan Hendrycks
admin_assigned
2024-03-26T09:19:39.622Z
63f55aa1b51da4d61da9c96b
/avatars/56cf9c2d8295c4549248d3b0a4933043.svg
Dan Hendrycks
user
hendrycks
66024c5f73c7ce480391b86f
Dawn Song
66024c5f73c7ce480391b870
Zhangyang Wang
66024c5f73c7ce480391b871
Bo Li
admin_assigned
2024-03-26T09:20:48.218Z
6493236b70d925ae8050a1bf
/avatars/b16069de1445cfa8608567175deaa2ae.svg
Bo Li
user
BoLi-aisecure
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient
  LLMs Under Compression
Compressing high-capability Large Language Models (LLMs) has emerged as a
favored strategy for resource-efficient inferences. While state-of-the-art
(SoTA) compression methods boast impressive advancements in preserving benign
task performance, the potential risks of compression in terms of safety and
trustworthiness have been largely neglected. This study conducts the first,
thorough evaluation of three (3) leading LLMs using five (5) SoTA compression
techniques across eight (8) trustworthiness dimensions. Our experiments
highlight the intricate interplay between compression and trustworthiness,
revealing some interesting patterns. We find that quantization is currently a
more effective approach than pruning in achieving efficiency and
trustworthiness simultaneously. For instance, a 4-bit quantized model retains
the trustworthiness of its original counterpart, but model pruning
significantly degrades trustworthiness, even at 50% sparsity. Moreover,
employing quantization within a moderate bit range could unexpectedly improve
certain trustworthiness dimensions such as ethics and fairness. Conversely,
extreme quantization to very low bit levels (3 bits) tends to significantly
reduce trustworthiness. This increased risk cannot be uncovered by looking at
benign performance alone, in turn, mandating comprehensive trustworthiness
evaluation in practice. These findings culminate in practical recommendations
for simultaneously achieving high utility, efficiency, and trustworthiness in
LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
Compression techniques, particularly quantization, have varying impacts on the trustworthiness of large language models, with quantization within moderate bit ranges enhancing certain ethical dimensions while extreme quantization degrades trustworthiness.
Large Language Models LLMs compression techniques quantization pruning trustworthiness ethics fairness