RADLADS: Rapid Attention Distillation to Linear Attention Decoders at
  Scale
We present Rapid Attention Distillation to Linear Attention Decoders at Scale
(RADLADS), a protocol for rapidly converting softmax attention transformers
into linear attention decoder models, along with two new RWKV-variant
architectures, and models converted from popular Qwen2.5 open source models in
7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens,
less than 0.005% of the token count used to train the original teacher models.
Converting to our 72B linear attention model costs less than \$2,000 USD at
today's prices, yet quality at inference remains close to the original
transformer. These models achieve state-of-the-art downstream performance
across a set of standard benchmarks for linear attention models of their size.
We release all our models on HuggingFace under the Apache 2.0 license, with the
exception of our 72B models which are also governed by the Qwen License
Agreement.
  Models at
https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102
Training Code at https://github.com/recursal/RADLADS-paper
681ac8a09f4ed2ece10fac18
Daniel Goldstein
claimed_verified
2025-05-07T08:08:53.268Z
647f4bac45baf21ad709fcd0
/avatars/14c04cdda95de676aeefa9ae3e7c19ba.svg
Dan Goldstein
user
SmerkyG
681ac8a09f4ed2ece10fac19
Eric Alcaide
681ac8a09f4ed2ece10fac1a
Janna Lu
claimed_verified
2025-05-08T08:59:35.127Z
66b3d98e040c500914ef558f
/avatars/a90f8306dbd7747520ce5b941ee3bbcb.svg
Janna
user
jannalu
681ac8a09f4ed2ece10fac1b
Eugene Cheah
RADLADS: Rapid Attention Distillation to Linear Attention Decoders at
  Scale
We present Rapid Attention Distillation to Linear Attention Decoders at Scale
(RADLADS), a protocol for rapidly converting softmax attention transformers
into linear attention decoder models, along with two new RWKV-variant
architectures, and models converted from popular Qwen2.5 open source models in
7B, 32B, and 72B sizes. Our conversion process requires only 350-700M tokens,
less than 0.005% of the token count used to train the original teacher models.
Converting to our 72B linear attention model costs less than \$2,000 USD at
today's prices, yet quality at inference remains close to the original
transformer. These models achieve state-of-the-art downstream performance
across a set of standard benchmarks for linear attention models of their size.
We release all our models on HuggingFace under the Apache 2.0 license, with the
exception of our 72B models which are also governed by the Qwen License
Agreement.
  Models at
https://huggingface.co/collections/recursal/radlads-6818ee69e99e729ba8a87102
Training Code at https://github.com/recursal/RADLADS-paper
A protocol converts softmax attention transformers into linear attention decoders using minimal tokens while maintaining quality and performance.
softmax attention transformers linear attention decoders Rapid Attention Distillation RADLADS RWKV-variant architectures Qwen2.5 models HuggingFace Apache 2.0 license Qwen License Agreement