TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
  Sequence Processing
MEGA is a recent transformer-based architecture, which utilizes a linear
recurrent operator whose parallel computation, based on the FFT, scales as
O(LlogL), with L being the sequence length. We build upon their approach by
replacing the linear recurrence with a special temporal convolutional network
which permits larger receptive field size with shallower networks, and reduces
the computational complexity to O(L). The resulting model is called TCNCA, a
Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
outperforms MEGA, reaching a lower loss with 1.37times/1.24times faster
forward/backward pass during training. The dilated convolutions used in TCNCA
are consistently and significantly faster operations than the FFT-based
parallelized recurrence in GPUs, making them a scalable candidate for handling
very large sequence lengths: they are up to 7.07times/2.86times faster in
the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
achieves, on average, 1.28times speed-up during inference with similar
accuracy to what MEGA achieves. On associative recall, we find that even a
simplified version of TCNCA, without excessive multiplicative and additive
interactions, remains superior or competitive to MEGA on a range of sequence
lengths and vocabulary sizes.
6577daa276b6de7978f2e9ae
Aleksandar Terzic
6577daa276b6de7978f2e9af
Michael Hersche
6577daa276b6de7978f2e9b0
Geethan Karunaratne
6577daa276b6de7978f2e9b1
Luca Benini
admin_assigned
2023-12-12T09:09:19.664Z
64a6a9370602161aa9090ecd
/avatars/8488546102fefb5c5b218469e6f886c0.svg
Luca Benini
user
lbenini
6577daa276b6de7978f2e9b2
Abu Sebastian
6577daa276b6de7978f2e9b3
Abbas Rahimi
TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
  Sequence Processing
MEGA is a recent transformer-based architecture, which utilizes a linear
recurrent operator whose parallel computation, based on the FFT, scales as
O(LlogL), with L being the sequence length. We build upon their approach by
replacing the linear recurrence with a special temporal convolutional network
which permits larger receptive field size with shallower networks, and reduces
the computational complexity to O(L). The resulting model is called TCNCA, a
Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
outperforms MEGA, reaching a lower loss with 1.37times/1.24times faster
forward/backward pass during training. The dilated convolutions used in TCNCA
are consistently and significantly faster operations than the FFT-based
parallelized recurrence in GPUs, making them a scalable candidate for handling
very large sequence lengths: they are up to 7.07times/2.86times faster in
the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
achieves, on average, 1.28times speed-up during inference with similar
accuracy to what MEGA achieves. On associative recall, we find that even a
simplified version of TCNCA, without excessive multiplicative and additive
interactions, remains superior or competitive to MEGA on a range of sequence
lengths and vocabulary sizes.
A Temporal Convolutional Network with Chunked Attention (TCNCA) outperforms MEGA by reducing computational complexity and maintaining or improving accuracy across various tasks.
transformer-based architecture linear recurrent operator FFT parallel computation temporal convolutional network receptive field size computational complexity TCNCA chunked attention EnWik8 long-range-arena associative recall dilated convolutions speed-up inference