HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide
  Resolution
Genomic (DNA) sequences encode an enormous amount of information for gene
regulation and protein synthesis. Similar to natural language models,
researchers have proposed foundation models in genomics to learn generalizable
features from unlabeled genome data that can then be fine-tuned for downstream
tasks such as identifying regulatory elements. Due to the quadratic scaling of
attention, previous Transformer-based genomic models have used 512 to 4k tokens
as context (<0.001% of the human genome), significantly limiting the modeling
of long-range interactions in DNA. In addition, these methods rely on
tokenizers to aggregate meaningful DNA units, losing single nucleotide
resolution where subtle genetic variations can completely alter protein
function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large
language model based on implicit convolutions was shown to match attention in
quality while allowing longer context lengths and lower time complexity.
Leveraging Hyenas new long-range capabilities, we present HyenaDNA, a genomic
foundation model pretrained on the human reference genome with context lengths
of up to 1 million tokens at the single nucleotide-level, an up to 500x
increase over previous dense attention-based models. HyenaDNA scales
sub-quadratically in sequence length (training up to 160x faster than
Transformer), uses single nucleotide tokens, and has full global context at
each layer. We explore what longer context enables - including the first use of
in-context learning in genomics for simple adaptation to novel tasks without
updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide
Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 17 datasets
using a model with orders of magnitude less parameters and pretraining data. On
the GenomicBenchmarks, HyenaDNA surpasses SotA on all 8 datasets on average by
+9 accuracy points.
649cd97cc1ae48cf4d8b85aa
Eric Nguyen
claimed_verified
2023-06-29T20:55:46.401Z
63f87f8ada440a47e9f5b7f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63f87f8ada440a47e9f5b7f0/LE4CUPwT6khOU-gib520a.jpeg
Eric Nguyen
user
exnx
649cd97cc1ae48cf4d8b85ab
Michael Poli
claimed_verified
2023-06-29T20:55:41.989Z
62a1306bbe7fa896d2c8de44
https://cdn-avatars.huggingface.co/v1/production/uploads/62a1306bbe7fa896d2c8de44/ndUBVtc5A3nsfUE47vood.png
Michael Poli
user
Zymrael
649cd97cc1ae48cf4d8b85ac
Marjan Faizi
649cd97cc1ae48cf4d8b85ad
Armin Thomas
649cd97cc1ae48cf4d8b85ae
Callum Birch-Sykes
649cd97cc1ae48cf4d8b85af
Michael Wornow
admin_assigned
2023-06-29T11:08:16.716Z
6025f42e0703121c822a80b4
/avatars/5a50df2c0618b49739bb6db57d1aa2b8.svg
Michael Wornow
user
Miking98
649cd97cc1ae48cf4d8b85b0
Aman Patel
649cd97cc1ae48cf4d8b85b1
Clayton Rabideau
649cd97cc1ae48cf4d8b85b2
Stefano Massaroli
649cd97cc1ae48cf4d8b85b3
Yoshua Bengio
649cd97cc1ae48cf4d8b85b4
Stefano Ermon
649cd97cc1ae48cf4d8b85b5
Stephen A. Baccus
649cd97cc1ae48cf4d8b85b6
Chris Ré
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide
  Resolution
Genomic (DNA) sequences encode an enormous amount of information for gene
regulation and protein synthesis. Similar to natural language models,
researchers have proposed foundation models in genomics to learn generalizable
features from unlabeled genome data that can then be fine-tuned for downstream
tasks such as identifying regulatory elements. Due to the quadratic scaling of
attention, previous Transformer-based genomic models have used 512 to 4k tokens
as context (<0.001% of the human genome), significantly limiting the modeling
of long-range interactions in DNA. In addition, these methods rely on
tokenizers to aggregate meaningful DNA units, losing single nucleotide
resolution where subtle genetic variations can completely alter protein
function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large
language model based on implicit convolutions was shown to match attention in
quality while allowing longer context lengths and lower time complexity.
Leveraging Hyenas new long-range capabilities, we present HyenaDNA, a genomic
foundation model pretrained on the human reference genome with context lengths
of up to 1 million tokens at the single nucleotide-level, an up to 500x
increase over previous dense attention-based models. HyenaDNA scales
sub-quadratically in sequence length (training up to 160x faster than
Transformer), uses single nucleotide tokens, and has full global context at
each layer. We explore what longer context enables - including the first use of
in-context learning in genomics for simple adaptation to novel tasks without
updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide
Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 17 datasets
using a model with orders of magnitude less parameters and pretraining data. On
the GenomicBenchmarks, HyenaDNA surpasses SotA on all 8 datasets on average by
+9 accuracy points.
HyenaDNA, a genomic foundation model based on implicit convolutions, achieves state-of-the-art performance on various benchmarks with longer context lengths, single nucleotide resolution, and fewer parameters.
Transformer implicit convolutions Hyena context lengths single nucleotide-level in-context learning Nucleotide Transformer GenomicBenchmarks