Tuning Language Models by Proxy
Despite the general capabilities of large pretrained language models, they
consistently benefit from further adaptation to better achieve desired
behaviors. However, tuning these models has become increasingly
resource-intensive, or impossible when model weights are private. We introduce
proxy-tuning, a lightweight decoding-time algorithm that operates on top of
black-box LMs to achieve the result of directly tuning the model, but by
accessing only its prediction over the output vocabulary. Our method instead
tunes a smaller LM, then applies the difference between the predictions of the
small tuned and untuned LMs to shift the original predictions of the base model
in the direction of tuning, while retaining the benefits of larger scale
pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using
proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its
truly-tuned chat version, when evaluated across knowledge, reasoning, and
safety benchmarks. Interestingly, when tested on TruthfulQA, proxy-tuned models
are actually more truthful than directly tuned models, possibly because
decoding-time guidance better retains the model's factual knowledge. We then
demonstrate the generality of proxy-tuning by applying it for domain adaptation
on code, and task-specific finetuning on question-answering and math problems.
Our work demonstrates the promise of using small tuned LMs to efficiently
customize large, potentially proprietary LMs through decoding-time guidance.
65a772455c58475cf91ca68d
Alisa Liu
admin_assigned
2024-01-17T09:42:45.304Z
62449da46b55ef458d430371
/avatars/b95954a88c7b15524a3d7776a4abe083.svg
Alisa Liu
user
alisawuffles
65a772455c58475cf91ca68e
Xiaochuang Han
admin_assigned
2024-01-17T09:42:51.194Z
635ea3969f24f6db0a1e2d0b
https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
Xiaochuang Han
user
xhan77
65a772455c58475cf91ca68f
Yizhong Wang
admin_assigned
2024-01-17T09:43:01.186Z
6269be67d1ac0cde592aba29
https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
Yizhong Wang
user
yizhongw
65a772455c58475cf91ca690
Yulia Tsvetkov
65a772455c58475cf91ca691
Yejin Choi
admin_assigned
2024-01-17T09:43:14.472Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
65a772455c58475cf91ca692
Noah A. Smith
Tuning Language Models by Proxy
Despite the general capabilities of large pretrained language models, they
consistently benefit from further adaptation to better achieve desired
behaviors. However, tuning these models has become increasingly
resource-intensive, or impossible when model weights are private. We introduce
proxy-tuning, a lightweight decoding-time algorithm that operates on top of
black-box LMs to achieve the result of directly tuning the model, but by
accessing only its prediction over the output vocabulary. Our method instead
tunes a smaller LM, then applies the difference between the predictions of the
small tuned and untuned LMs to shift the original predictions of the base model
in the direction of tuning, while retaining the benefits of larger scale
pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using
proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its
truly-tuned chat version, when evaluated across knowledge, reasoning, and
safety benchmarks. Interestingly, when tested on TruthfulQA, proxy-tuned models
are actually more truthful than directly tuned models, possibly because
decoding-time guidance better retains the model's factual knowledge. We then
demonstrate the generality of proxy-tuning by applying it for domain adaptation
on code, and task-specific finetuning on question-answering and math problems.
Our work demonstrates the promise of using small tuned LMs to efficiently
customize large, potentially proprietary LMs through decoding-time guidance.
Proxy-tuning, a lightweight decoding-time algorithm, customizes large language models efficiently using smaller LMs without accessing their weights, achieving competitive performance and better factual truthfulness.
proxy-tuning decoding-time algorithm black-box LMs prediction over the output vocabulary domain adaptation task-specific fine-tuning Llama2-70B TruthfulQA