BioMamba: A Pre-trained Biomedical Language Representation Model
  Leveraging Mamba
The advancement of natural language processing (NLP) in biology hinges on
models' ability to interpret intricate biomedical literature. Traditional
models often struggle with the complex and domain-specific language in this
field. In this paper, we present BioMamba, a pre-trained model specifically
designed for biomedical text mining. BioMamba builds upon the Mamba
architecture and is pre-trained on an extensive corpus of biomedical
literature. Our empirical studies demonstrate that BioMamba significantly
outperforms models like BioBERT and general-domain Mamba across various
biomedical tasks. For instance, BioMamba achieves a 100 times reduction in
perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test
set. We provide an overview of the model architecture, pre-training process,
and fine-tuning techniques. Additionally, we release the code and trained model
to facilitate further research.
66b1f44fba6dc98b702963ff
Ling Yue
claimed_verified
2024-08-08T07:41:51.097Z
66739484a660dfbb2643eb3d
/avatars/c9a6c6a1e295e448dacfccb1a2860e8a.svg
Leo Y
user
LeoYML
66b1f44fba6dc98b70296400
Sixue Xing
admin_assigned
2024-08-06T11:59:49.024Z
66217de1c92239f4997a4c75
/avatars/b1c4d74878b62ffdb07f65971b4f5501.svg
Sixue Xing
user
sixuexing
66b1f44fba6dc98b70296401
Yingzhou Lu
66b1f44fba6dc98b70296402
Tianfan Fu
admin_assigned
2024-08-06T11:59:36.574Z
6322429c7bb41a713daf40fb
/avatars/e226463a33ac51d7052eb6bc6d35bed5.svg
Tianfan Fu
user
TianfanFu
BioMamba: A Pre-trained Biomedical Language Representation Model
  Leveraging Mamba
The advancement of natural language processing (NLP) in biology hinges on
models' ability to interpret intricate biomedical literature. Traditional
models often struggle with the complex and domain-specific language in this
field. In this paper, we present BioMamba, a pre-trained model specifically
designed for biomedical text mining. BioMamba builds upon the Mamba
architecture and is pre-trained on an extensive corpus of biomedical
literature. Our empirical studies demonstrate that BioMamba significantly
outperforms models like BioBERT and general-domain Mamba across various
biomedical tasks. For instance, BioMamba achieves a 100 times reduction in
perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test
set. We provide an overview of the model architecture, pre-training process,
and fine-tuning techniques. Additionally, we release the code and trained model
to facilitate further research.
BioMamba, a pre-trained model based on Mamba architecture, excels in biomedical text mining, outperforming other models in perplexity and cross-entropy loss.
BioMamba Mamba architecture biomedical text mining pre-trained model BioBERT BioASQ test set perplexity cross-entropy loss