TokenFormer: Rethinking Transformer Scaling with Tokenized Model
  Parameters
Transformers have become the predominant architecture in foundation models
due to their excellent performance across various domains. However, the
substantial cost of scaling these models remains a significant concern. This
problem arises primarily from their dependence on a fixed number of parameters
within linear projections. When architectural modifications (e.g., channel
dimensions) are introduced, the entire model typically requires retraining from
scratch. As model sizes continue growing, this strategy results in increasingly
high computational costs and becomes unsustainable. To overcome this problem,
we introduce TokenFormer, a natively scalable architecture that leverages the
attention mechanism not only for computations among input tokens but also for
interactions between tokens and model parameters, thereby enhancing
architectural flexibility. By treating model parameters as tokens, we replace
all the linear projections in Transformers with our token-parameter attention
layer, where input tokens act as queries and model parameters as keys and
values. This reformulation allows for progressive and efficient scaling without
necessitating retraining from scratch. Our model scales from 124M to 1.4B
parameters by incrementally adding new key-value parameter pairs, achieving
performance comparable to Transformers trained from scratch while greatly
reducing training costs. Code and models are available at
https://github.com/Haiyang-W/TokenFormer.
6722db7db23a243a937ee098
Haiyang Wang
claimed_verified
2024-10-31T14:06:06.262Z
65f43c3cc9940817caaf4434
/avatars/ecec2856ba7a7d3421a2071a0a88800b.svg
Haiyang Wang
user
Haiyang-W
6722db7db23a243a937ee099
Yue Fan
6722db7db23a243a937ee09a
Muhammad Ferjad Naeem
claimed_verified
2025-02-26T15:37:48.498Z
67bf33a512368ec2fad4fe29
/avatars/ea5c03744ec1c2bcc0e6c13efc8f7ddc.svg
Muhammad Ferjad Naeem
user
ferjad
6722db7db23a243a937ee09b
Yongqin Xian
6722db7db23a243a937ee09c
Jan Eric Lenssen
6722db7db23a243a937ee09d
Liwei Wang
6722db7db23a243a937ee09e
Federico Tombari
6722db7db23a243a937ee09f
Bernt Schiele
TokenFormer: Rethinking Transformer Scaling with Tokenized Model
  Parameters
Transformers have become the predominant architecture in foundation models
due to their excellent performance across various domains. However, the
substantial cost of scaling these models remains a significant concern. This
problem arises primarily from their dependence on a fixed number of parameters
within linear projections. When architectural modifications (e.g., channel
dimensions) are introduced, the entire model typically requires retraining from
scratch. As model sizes continue growing, this strategy results in increasingly
high computational costs and becomes unsustainable. To overcome this problem,
we introduce TokenFormer, a natively scalable architecture that leverages the
attention mechanism not only for computations among input tokens but also for
interactions between tokens and model parameters, thereby enhancing
architectural flexibility. By treating model parameters as tokens, we replace
all the linear projections in Transformers with our token-parameter attention
layer, where input tokens act as queries and model parameters as keys and
values. This reformulation allows for progressive and efficient scaling without
necessitating retraining from scratch. Our model scales from 124M to 1.4B
parameters by incrementally adding new key-value parameter pairs, achieving
performance comparable to Transformers trained from scratch while greatly
reducing training costs. Code and models are available at
https://github.com/Haiyang-W/TokenFormer.
TokenFormer, a scalable architecture for Transformers, enhances flexibility by treating model parameters as tokens, allowing for efficient scaling without retraining.
Transformers token-parameter attention key-value parameter pairs