Clinical ModernBERT: An efficient and long context encoder for
  biomedical text
We introduce Clinical ModernBERT, a transformer based encoder pretrained on
large scale biomedical literature, clinical notes, and medical ontologies,
incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with
their textual descriptions. Building on ModernBERT the current state of the art
natural language text encoder featuring architectural upgrades such as rotary
positional embeddings (RoPE), Flash Attention, and extended context length up
to 8,192 tokens our model adapts these innovations specifically for biomedical
and clinical domains. Clinical ModernBERT excels at producing semantically rich
representations tailored for long context tasks. We validate this both by
analyzing its pretrained weights and through empirical evaluation on a
comprehensive suite of clinical NLP benchmarks.
67f4889c7e1624ebbaef710d
Simon A. Lee
claimed_verified
2025-04-08T06:51:44.271Z
652ee41e7b0079ff035b2269
https://cdn-avatars.huggingface.co/v1/production/uploads/652ee41e7b0079ff035b2269/jv-2-UPHN_e9Lhwl3Evwh.jpeg
Simon Lee
user
Simonlee711
67f4889c7e1624ebbaef710e
Anthony Wu
claimed_verified
2025-04-15T07:55:49.038Z
66ae56e72bc4568096d770df
/avatars/bcb255a24583c32719985dafc5f35e5a.svg
Anthony Wu
user
anthonytkwu
67f4889c7e1624ebbaef710f
Jeffrey N. Chiang
Clinical ModernBERT: An efficient and long context encoder for
  biomedical text
We introduce Clinical ModernBERT, a transformer based encoder pretrained on
large scale biomedical literature, clinical notes, and medical ontologies,
incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with
their textual descriptions. Building on ModernBERT the current state of the art
natural language text encoder featuring architectural upgrades such as rotary
positional embeddings (RoPE), Flash Attention, and extended context length up
to 8,192 tokens our model adapts these innovations specifically for biomedical
and clinical domains. Clinical ModernBERT excels at producing semantically rich
representations tailored for long context tasks. We validate this both by
analyzing its pretrained weights and through empirical evaluation on a
comprehensive suite of clinical NLP benchmarks.
Clinical ModernBERT, a transformer-based encoder pre-trained on extensive biomedical data, incorporates innovative features like rotary positional embeddings and Flash Attention, excelling in long-context clinical NLP tasks.
transformer based encoder rotary positional embeddings Flash Attention biomedical literature clinical notes medical ontologies PubMed abstracts MIMIC IV clinical data medical codes semantically rich representations clinical NLP benchmarks