A Unified View of Delta Parameter Editing in Post-Trained Large-Scale
  Models
Post-training has emerged as a crucial paradigm for adapting large-scale
pre-trained models to various tasks, whose effects are fully reflected by delta
parameters (i.e., the disparity between post-trained and pre-trained
parameters). While numerous studies have explored delta parameter properties
via operations like pruning, quantization, low-rank approximation, and
extrapolation, a unified framework for systematically examining these
characteristics has been lacking. In this paper, we propose a novel perspective
based on Riemann sum approximation of the loss function to elucidate delta
parameter editing operations. Our analysis categorizes existing methods into
three classes based on their post-editing performance: competitive, decreased,
and improved, explaining how they are expressed by the Riemann sum
approximation term and how they alter the model performance. Extensive
experiments on both visual and language models, including ViT, LLaMA 3, Qwen 2,
and Mistral, corroborate our theoretical findings. Furthermore, we introduce
extensions to existing techniques like DARE and BitDelta, highlighting their
limitations in leveraging the properties of delta parameters and reorganizing
them into general expressions to enhance the applicability and effectiveness of
delta parameter editing in post-trained models.
6711c61c1e98ec0ba6bc418c
Qiaoyu Tang
claimed_verified
2024-10-18T08:12:33.542Z
64ed96f3067fbb625f83766a
/avatars/e8794ba19bdd965615bd9ead7df77d7d.svg
Qiaoyu Tang
user
TangQiaoYu
6711c61c1e98ec0ba6bc418d
Le Yu
claimed_verified
2025-05-18T08:48:14.285Z
6455056fe4952d1c6cb45e49
/avatars/bbdcd4405497f11ff3774352277d1ec7.svg
Le Yu
user
vanillaOVO
6711c61c1e98ec0ba6bc418e
Bowen Yu
admin_assigned
2024-10-18T12:36:39.144Z
6438b43ab2ea24b52ebac2b9
/avatars/84133cd719a4b1e2f5c1a74178425f86.svg
Bowen Yu
user
bwy
6711c61c1e98ec0ba6bc418f
Hongyu Lin
admin_assigned
2024-10-18T12:36:25.734Z
6711c702f858a456b4b9f3a4
/avatars/178e9567c3111ab22717c3c0dd003a6a.svg
Hongyu  Lin
user
sanmusunrise
6711c61c1e98ec0ba6bc4190
Keming Lu
admin_assigned
2024-10-18T12:36:53.561Z
6453fa96ed6d7fede94408e0
/avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
Keming Lu
user
keminglu
6711c61c1e98ec0ba6bc4191
Yaojie Lu
admin_assigned
2024-10-18T12:36:59.079Z
6216496a9b34d2fb49144599
https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
Yaojie Lu
user
luyaojie
6711c61c1e98ec0ba6bc4192
Xianpei Han
admin_assigned
2024-10-18T12:36:19.025Z
65e99a77e71555ed193609cf
/avatars/38ceb127883944677665da967d17dd18.svg
Xianpei Han
user
xphan
6711c61c1e98ec0ba6bc4193
Le Sun
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale
  Models
Post-training has emerged as a crucial paradigm for adapting large-scale
pre-trained models to various tasks, whose effects are fully reflected by delta
parameters (i.e., the disparity between post-trained and pre-trained
parameters). While numerous studies have explored delta parameter properties
via operations like pruning, quantization, low-rank approximation, and
extrapolation, a unified framework for systematically examining these
characteristics has been lacking. In this paper, we propose a novel perspective
based on Riemann sum approximation of the loss function to elucidate delta
parameter editing operations. Our analysis categorizes existing methods into
three classes based on their post-editing performance: competitive, decreased,
and improved, explaining how they are expressed by the Riemann sum
approximation term and how they alter the model performance. Extensive
experiments on both visual and language models, including ViT, LLaMA 3, Qwen 2,
and Mistral, corroborate our theoretical findings. Furthermore, we introduce
extensions to existing techniques like DARE and BitDelta, highlighting their
limitations in leveraging the properties of delta parameters and reorganizing
them into general expressions to enhance the applicability and effectiveness of
delta parameter editing in post-trained models.
A unified framework based on Riemann sum approximation categorizes and analyzes delta parameter editing operations in post-trained models, enhancing their applicability and effectiveness.
Riemann sum approximation delta parameters pruning quantization low-rank approximation extrapolation competitive decreased improved ViT LLaMA 3 Qwen 2 Mistral DARE BitDelta