Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Large-scale sequence modeling has sparked rapid advances that now extend into
biology and genomics. However, modeling genomic sequences introduces challenges
such as the need to model long-range token interactions, the effects of
upstream and downstream regions of the genome, and the reverse complementarity
(RC) of DNA. Here, we propose an architecture motivated by these challenges
that builds off the long-range Mamba block, and extends it to a BiMamba
component that supports bi-directionality, and to a MambaDNA block that
additionally supports RC equivariance. We use MambaDNA as the basis of
Caduceus, the first family of RC equivariant bi-directional long-range DNA
language models, and we introduce pre-training and fine-tuning strategies that
yield Caduceus DNA foundation models. Caduceus outperforms previous long-range
models on downstream benchmarks; on a challenging long-range variant effect
prediction task, Caduceus exceeds the performance of 10x larger models that do
not leverage bi-directionality or equivariance.
65e923a00ba2a33b5a7da86f
Yair Schiff
admin_assigned
2024-03-07T10:11:54.031Z
6418e5e722270b3ccf15dfef
https://cdn-avatars.huggingface.co/v1/production/uploads/6418e5e722270b3ccf15dfef/ZiammTpL9FVMuE7DN8n5B.jpeg
Yair Schiff
user
yairschiff
65e923a00ba2a33b5a7da870
Chia-Hsiang Kao
admin_assigned
2024-03-07T10:12:10.628Z
6501eff07c3f4b17b897fa0b
/avatars/1e47934d03927c6e9b549083d7e7df7e.svg
Chia-Hsiang Kao
user
AaronKao
65e923a00ba2a33b5a7da871
Aaron Gokaslan
admin_assigned
2024-03-07T10:12:17.121Z
620573d0522e40b4a18d8763
/avatars/9353c064ef8ccac84d0397411d38fa90.svg
Aaron Gokaslan
user
Skylion007
65e923a00ba2a33b5a7da872
Tri Dao
admin_assigned
2024-03-07T10:12:24.009Z
64b8a6b5cf14c2fabe98159b
/avatars/dbc009451865435bf290791beadc4723.svg
Tri Dao
user
tridao
65e923a00ba2a33b5a7da873
Albert Gu
admin_assigned
2024-03-07T10:12:39.524Z
642619b62864ea9e59de834e
/avatars/a293aac4aa53800ad7dd08b28bd6b6f5.svg
Albert Gu
user
albertgu
65e923a00ba2a33b5a7da874
Volodymyr Kuleshov
admin_assigned
2024-03-07T10:12:47.046Z
640ea1c43c82bd463ee80e19
/avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
Volodymyr Kuleshov
user
kuleshov
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Large-scale sequence modeling has sparked rapid advances that now extend into
biology and genomics. However, modeling genomic sequences introduces challenges
such as the need to model long-range token interactions, the effects of
upstream and downstream regions of the genome, and the reverse complementarity
(RC) of DNA. Here, we propose an architecture motivated by these challenges
that builds off the long-range Mamba block, and extends it to a BiMamba
component that supports bi-directionality, and to a MambaDNA block that
additionally supports RC equivariance. We use MambaDNA as the basis of
Caduceus, the first family of RC equivariant bi-directional long-range DNA
language models, and we introduce pre-training and fine-tuning strategies that
yield Caduceus DNA foundation models. Caduceus outperforms previous long-range
models on downstream benchmarks; on a challenging long-range variant effect
prediction task, Caduceus exceeds the performance of 10x larger models that do
not leverage bi-directionality or equivariance.
A new family of DNA language models, Caduceus, supports bi-directionality and reverse complementarity, achieving superior performance on long-range genomic sequence modeling tasks.
sequence modeling genomic sequences long-range token interactions reverse complementarity Mamba block BiMamba MambaDNA RC equivariant long-range DNA language models pre-training fine-tuning variant effect prediction