Emergent properties with repeated examples
We study the performance of transformers as a function of the number of
repetitions of training examples with algorithmically generated datasets. On
three problems of mathematics: the greatest common divisor, modular
multiplication, and matrix eigenvalues, we show that for a fixed number of
training steps, models trained on smaller sets of repeated examples outperform
models trained on larger sets of single-use examples. We also demonstrate that
two-set training - repeated use of a small random subset of examples, along
normal sampling on the rest of the training set - provides for faster learning
and better performance. This highlights that the benefits of repetition can
outweigh those of data diversity. These datasets and problems provide a
controlled setting to shed light on the still poorly understood interplay
between generalization and memorization in deep learning.
67088ce7886c2765f184aebe
François Charton
67088ce7886c2765f184aebf
Julia Kempe
admin_assigned
2024-10-11T11:08:47.955Z
65ce30e06da01df536eded5a
/avatars/04c32cba7a3bbaf9ea5dee88c96cf87b.svg
Julia Kempe
user
Knykny
Emergent properties with repeated examples
We study the performance of transformers as a function of the number of
repetitions of training examples with algorithmically generated datasets. On
three problems of mathematics: the greatest common divisor, modular
multiplication, and matrix eigenvalues, we show that for a fixed number of
training steps, models trained on smaller sets of repeated examples outperform
models trained on larger sets of single-use examples. We also demonstrate that
two-set training - repeated use of a small random subset of examples, along
normal sampling on the rest of the training set - provides for faster learning
and better performance. This highlights that the benefits of repetition can
outweigh those of data diversity. These datasets and problems provide a
controlled setting to shed light on the still poorly understood interplay
between generalization and memorization in deep learning.
Transformers trained on repeated algorithmically generated datasets perform better than those trained on a diversity of examples, highlighting the benefits of repetition in learning.
transformers algorithmically generated datasets greatest common divisor modular multiplication matrix eigenvalues two-set training repetition generalization memorization