Instruction Following without Instruction Tuning
Instruction tuning commonly means finetuning a language model on
instruction-response pairs. We discover two forms of adaptation (tuning) that
are deficient compared to instruction tuning, yet still yield instruction
following; we call this implicit instruction tuning. We first find that
instruction-response pairs are not necessary: training solely on responses,
without any corresponding instructions, yields instruction following. This
suggests pretrained models have an instruction-response mapping which is
revealed by teaching the model the desired distribution of responses. However,
we then find it's not necessary to teach the desired distribution of responses:
instruction-response training on narrow-domain data like poetry still leads to
broad instruction-following behavior like recipe generation. In particular,
when instructions are very different from those in the narrow finetuning
domain, models' responses do not adhere to the style of the finetuning domain.
To begin to explain implicit instruction tuning, we hypothesize that very
simple changes to a language model's distribution yield instruction following.
We support this by hand-writing a rule-based language model which yields
instruction following in a product-of-experts with a pretrained model. The
rules are to slowly increase the probability of ending the sequence, penalize
repetition, and uniformly change 15 words' probabilities. In summary,
adaptations made without being designed to yield instruction following can do
so implicitly.
66f69c7519fa4bbc452507bb
John Hewitt
admin_assigned
2024-10-01T13:36:25.646Z
646ec1161a198f8520f3c705
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QPCRraySl9zTnf7eG5ZJk.png
John Hewitt
user
johnhew
66f69c7519fa4bbc452507bc
Nelson F. Liu
66f69c7519fa4bbc452507bd
Percy Liang
admin_assigned
2024-10-01T13:35:42.185Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
66f69c7519fa4bbc452507be
Christopher D. Manning
admin_assigned
2024-10-01T13:35:34.664Z
5f555a1f8bf55658acfed19e
https://cdn-avatars.huggingface.co/v1/production/uploads/1628695407761-5f555a1f8bf55658acfed19e.jpeg
Christopher Manning
user
manning
Instruction Following without Instruction Tuning
Instruction tuning commonly means finetuning a language model on
instruction-response pairs. We discover two forms of adaptation (tuning) that
are deficient compared to instruction tuning, yet still yield instruction
following; we call this implicit instruction tuning. We first find that
instruction-response pairs are not necessary: training solely on responses,
without any corresponding instructions, yields instruction following. This
suggests pretrained models have an instruction-response mapping which is
revealed by teaching the model the desired distribution of responses. However,
we then find it's not necessary to teach the desired distribution of responses:
instruction-response training on narrow-domain data like poetry still leads to
broad instruction-following behavior like recipe generation. In particular,
when instructions are very different from those in the narrow finetuning
domain, models' responses do not adhere to the style of the finetuning domain.
To begin to explain implicit instruction tuning, we hypothesize that very
simple changes to a language model's distribution yield instruction following.
We support this by hand-writing a rule-based language model which yields
instruction following in a product-of-experts with a pretrained model. The
rules are to slowly increase the probability of ending the sequence, penalize
repetition, and uniformly change 15 words' probabilities. In summary,
adaptations made without being designed to yield instruction following can do
so implicitly.
Adaptations without explicit instruction tuning, such as training on responses or narrow-domain instruction-response pairs, can still lead to broad instruction-following behavior in language models.
instruction tuning implicit instruction tuning instruction-response pairs distribution of responses narrow-domain data broad instruction-following rule-based language model product-of-experts probability of ending penalize repetition