FLARE: Fast Low-rank Attention Routing Engine
The quadratic complexity of self-attention limits its applicability and
scalability on large unstructured meshes. We introduce Fast Low-rank Attention
Routing Engine (FLARE), a linear complexity self-attention mechanism that
routes attention through fixed-length latent sequences. Each attention head
performs global communication among N tokens by projecting the input sequence
onto a fixed length latent sequence of M ll N tokens using learnable query
tokens. By routing attention through a bottleneck sequence, FLARE learns a
low-rank form of attention that can be applied at O(NM) cost. FLARE not only
scales to unprecedented problem sizes, but also delivers superior accuracy
compared to state-of-the-art neural PDE surrogates across diverse benchmarks.
We also release a new additive manufacturing dataset to spur further research.
Our code is available at https://github.com/vpuri3/FLARE.py.
68a4b57e6cf0bf898542eb76
Vedant Puri
claimed_verified
2025-08-20T08:52:19.021Z
6729087b934501c4f242f768
https://cdn-avatars.huggingface.co/v1/production/uploads/6729087b934501c4f242f768/u4KZEqXovyq3uZJDnK-aX.jpeg
Vedant Puri
user
vedantpuri
68a4b57e6cf0bf898542eb77
Aditya Joglekar
68a4b57e6cf0bf898542eb78
Kevin Ferguson
68a4b57e6cf0bf898542eb79
Yu-hsuan Chen
68a4b57e6cf0bf898542eb7a
Yongjie Jessica Zhang
68a4b57e6cf0bf898542eb7b
Levent Burak Kara
FLARE: Fast Low-rank Attention Routing Engine
The quadratic complexity of self-attention limits its applicability and
scalability on large unstructured meshes. We introduce Fast Low-rank Attention
Routing Engine (FLARE), a linear complexity self-attention mechanism that
routes attention through fixed-length latent sequences. Each attention head
performs global communication among N tokens by projecting the input sequence
onto a fixed length latent sequence of M ll N tokens using learnable query
tokens. By routing attention through a bottleneck sequence, FLARE learns a
low-rank form of attention that can be applied at O(NM) cost. FLARE not only
scales to unprecedented problem sizes, but also delivers superior accuracy
compared to state-of-the-art neural PDE surrogates across diverse benchmarks.
We also release a new additive manufacturing dataset to spur further research.
Our code is available at https://github.com/vpuri3/FLARE.py.
FLARE, a linear complexity self-attention mechanism, improves scalability and accuracy for large unstructured meshes and neural PDE surrogates.
self-attention Fast Low-rank Attention Routing Engine FLARE linear complexity fixed-length latent sequences attention heads global communication learnable query tokens low-rank form of attention neural PDE surrogates additive manufacturing dataset