Don't Throw Away Your Pretrained Model
Alignment training has tradeoffs: it helps language models (LMs) gain in
reasoning and instruction following but might lose out on skills such as
creativity and calibration, where unaligned base models are better at. We aim
to make the best of both worlds through model collaboration, where different
models in the training pipeline collaborate and complement each other. Since LM
responses feature interleaving skills that favor different models, we propose
Switch Generation, where pretrained and aligned model versions take turns to
``speak'' in a response sequence. Specifically, we train a switcher LM by
learning from outcomes of choosing different models to generate the next
segment across diverse queries and contexts. At inference time, the switcher LM
guides different model checkpoints to dynamically generate the next segment
where their strengths are most needed. Extensive experiments with 8 model
collaboration baselines and 18 datasets show that 1) model collaboration
consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch
Generation further outperforms baselines by 12.9% on average. Further analysis
reveals that Switch Generation discovers compositional skills to solve problems
where individual models struggle and generalizes to unseen models and tasks,
reusing and repurposing by-products in expensive model training pipelines that
are otherwise discarded.
68f1666e6e0bef323a68fb28
Shangbin Feng
68f1666e6e0bef323a68fb29
Wenhao Yu
68f1666e6e0bef323a68fb2a
Yike Wang
68f1666e6e0bef323a68fb2b
Hongming Zhang
68f1666e6e0bef323a68fb2c
Yulia Tsvetkov
68f1666e6e0bef323a68fb2d
Dong Yu
Don't Throw Away Your Pretrained Model
Alignment training has tradeoffs: it helps language models (LMs) gain in
reasoning and instruction following but might lose out on skills such as
creativity and calibration, where unaligned base models are better at. We aim
to make the best of both worlds through model collaboration, where different
models in the training pipeline collaborate and complement each other. Since LM
responses feature interleaving skills that favor different models, we propose
Switch Generation, where pretrained and aligned model versions take turns to
``speak'' in a response sequence. Specifically, we train a switcher LM by
learning from outcomes of choosing different models to generate the next
segment across diverse queries and contexts. At inference time, the switcher LM
guides different model checkpoints to dynamically generate the next segment
where their strengths are most needed. Extensive experiments with 8 model
collaboration baselines and 18 datasets show that 1) model collaboration
consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch
Generation further outperforms baselines by 12.9% on average. Further analysis
reveals that Switch Generation discovers compositional skills to solve problems
where individual models struggle and generalizes to unseen models and tasks,
reusing and repurposing by-products in expensive model training pipelines that
are otherwise discarded.
Model collaboration and Switch Generation improve language model performance by leveraging the strengths of different models in a dynamic and complementary manner.
alignment training language models reasoning instruction following creativity calibration model collaboration Switch Generation pretrained models aligned models switcher LM inference time compositional skills unseen models tasks expensive model training pipelines