Linear Transformers are Versatile In-Context Learners
Recent research has demonstrated that transformers, particularly linear
attention models, implicitly execute gradient-descent-like algorithms on data
provided in-context during their forward inference step. However, their
capability in handling more complex problems remains unexplored. In this paper,
we prove that any linear transformer maintains an implicit linear model and can
be interpreted as performing a variant of preconditioned gradient descent. We
also investigate the use of linear transformers in a challenging scenario where
the training data is corrupted with different levels of noise. Remarkably, we
demonstrate that for this problem linear transformers discover an intricate and
highly effective optimization algorithm, surpassing or matching in performance
many reasonable baselines. We reverse-engineer this algorithm and show that it
is a novel approach incorporating momentum and adaptive rescaling based on
noise levels. Our findings show that even linear transformers possess the
surprising ability to discover sophisticated optimization strategies.
65d821412c9f76a077ef2b23
Max Vladymyrov
admin_assigned
2024-02-23T13:18:02.726Z
6502fe0bb1792803da806d42
/avatars/60280ce59f1f0d67e4210b0453039282.svg
Max Vladymyrov
user
gozzo87
65d821412c9f76a077ef2b24
Johannes von Oswald
admin_assigned
2024-02-23T13:18:09.984Z
6502de229421d71875f52170
/avatars/cff4c35bb8f5704d47bd8a74e7f605e6.svg
Johannes von Oswald
user
voswaldj
65d821412c9f76a077ef2b25
Mark Sandler
65d821412c9f76a077ef2b26
Rong Ge
Linear Transformers are Versatile In-Context Learners
Recent research has demonstrated that transformers, particularly linear
attention models, implicitly execute gradient-descent-like algorithms on data
provided in-context during their forward inference step. However, their
capability in handling more complex problems remains unexplored. In this paper,
we prove that any linear transformer maintains an implicit linear model and can
be interpreted as performing a variant of preconditioned gradient descent. We
also investigate the use of linear transformers in a challenging scenario where
the training data is corrupted with different levels of noise. Remarkably, we
demonstrate that for this problem linear transformers discover an intricate and
highly effective optimization algorithm, surpassing or matching in performance
many reasonable baselines. We reverse-engineer this algorithm and show that it
is a novel approach incorporating momentum and adaptive rescaling based on
noise levels. Our findings show that even linear transformers possess the
surprising ability to discover sophisticated optimization strategies.
Linear transformers are shown to perform a variant of preconditioned gradient descent and discover noisy data optimization strategies that outperform baselines.
linear transformers gradient descent preconditioned gradient descent momentum adaptive rescaling