In-Context Editing: Learning Knowledge from Self-Induced Distributions
The existing fine-tuning paradigm for language models is brittle in knowledge
editing scenarios, where the model must incorporate new information without
extensive retraining. This brittleness often results in overfitting, reduced
performance, and unnatural language generation. To address this, we propose
Consistent In-Context Editing (ICE), a novel approach that leverages the
model's in-context learning capability to tune toward a contextual distribution
rather than a one-hot target. ICE introduces a straightforward optimization
framework that includes both a target and a procedure, enhancing the robustness
and effectiveness of gradient-based tuning methods. We provide analytical
insights into ICE across four critical aspects of knowledge editing: accuracy,
locality, generalization, and linguistic quality, showing its advantages.
Experimental results across four datasets confirm the effectiveness of ICE and
demonstrate its potential for continual editing, ensuring that updated
information is incorporated while preserving the integrity of the model.
66719193202196a178f8d501
Siyuan Qi
claimed_verified
2024-06-18T14:13:55.628Z
6483ec5fe2f6f33be175c5c3
/avatars/6372d1139c65ed976e5e558654821649.svg
Siyuan Qi
user
syqi
66719193202196a178f8d502
Bangcheng Yang
claimed_verified
2024-07-02T07:39:42.252Z
63ece843c8827dd0f0f7b8c4
/avatars/e6c5840510ab88b007183cf79ed1bdf9.svg
Bangcheng Yang
user
dumbmouse
66719193202196a178f8d503
Kailin Jiang
claimed_verified
2024-06-19T20:21:01.784Z
65745569839aa08899ea5d27
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg
kailinjiang
user
kailinjiang
66719193202196a178f8d504
Xiaobo Wang
claimed_verified
2024-06-19T20:20:59.856Z
65e933fcacb29a596cd6eb6e
https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg
Xiaobo Wang
user
Yofuria
66719193202196a178f8d505
Jiaqi Li
66719193202196a178f8d506
Yifan Zhong
66719193202196a178f8d507
Yaodong Yang
66719193202196a178f8d508
Zilong Zheng
claimed_verified
2024-06-19T07:39:17.582Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
In-Context Editing: Learning Knowledge from Self-Induced Distributions
The existing fine-tuning paradigm for language models is brittle in knowledge
editing scenarios, where the model must incorporate new information without
extensive retraining. This brittleness often results in overfitting, reduced
performance, and unnatural language generation. To address this, we propose
Consistent In-Context Editing (ICE), a novel approach that leverages the
model's in-context learning capability to tune toward a contextual distribution
rather than a one-hot target. ICE introduces a straightforward optimization
framework that includes both a target and a procedure, enhancing the robustness
and effectiveness of gradient-based tuning methods. We provide analytical
insights into ICE across four critical aspects of knowledge editing: accuracy,
locality, generalization, and linguistic quality, showing its advantages.
Experimental results across four datasets confirm the effectiveness of ICE and
demonstrate its potential for continual editing, ensuring that updated
information is incorporated while preserving the integrity of the model.
Consistent In-Context Editing leverages in-context learning to robustly fine-tune language models, improving knowledge editing by maintaining accuracy, locality, generalization, and linguistic quality.
fine-tuning knowledge editing Consistent In-Context Editing in-context learning gradient-based tuning