Time is Encoded in the Weights of Finetuned Language Models
We present time vectors, a simple tool to customize language models to new
time periods. Time vectors are created by finetuning a language model on data
from a single time (e.g., a year or month), and then subtracting the weights of
the original pretrained model. This vector specifies a direction in weight
space that, as our experiments show, improves performance on text from that
time period. Time vectors specialized to adjacent time periods appear to be
positioned closer together in a manifold. Using this structure, we interpolate
between time vectors to induce new models that perform better on intervening
and future time periods, without any additional training. We demonstrate the
consistency of our findings across different tasks, domains, model sizes, and
time scales. Our results suggest that time is encoded in the weight space of
finetuned models.
65850bd6b476868782d5379a
Kai Nylund
admin_assigned
2023-12-22T11:03:03.126Z
6260b95e4fe7f6521dd6a083
/avatars/f3e20c204b83199507646604a30660bb.svg
Kai Nylund
user
KaiNylund
65850bd6b476868782d5379b
Suchin Gururangan
admin_assigned
2023-12-22T11:03:10.681Z
5e9272254957053f60648a01
https://cdn-avatars.huggingface.co/v1/production/uploads/1586657211685-5e9272254957053f60648a01.png
Suchin Gururangan
user
suching
65850bd6b476868782d5379c
Noah A. Smith
Time is Encoded in the Weights of Finetuned Language Models
We present time vectors, a simple tool to customize language models to new
time periods. Time vectors are created by finetuning a language model on data
from a single time (e.g., a year or month), and then subtracting the weights of
the original pretrained model. This vector specifies a direction in weight
space that, as our experiments show, improves performance on text from that
time period. Time vectors specialized to adjacent time periods appear to be
positioned closer together in a manifold. Using this structure, we interpolate
between time vectors to induce new models that perform better on intervening
and future time periods, without any additional training. We demonstrate the
consistency of our findings across different tasks, domains, model sizes, and
time scales. Our results suggest that time is encoded in the weight space of
finetuned models.
Time vectors, derived by fine-tuning language models on specific time periods, enhance performance on text from those periods and can be interpolated to improve performance on future periods without further training.
time vectors finetuning weight space interpolation performance language models time periods manifold weight space time encoding