BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance
on a wide variety of biomedical NLP tasks. However, these models have hundreds
of billions of parameters, are computationally expensive to run, require users
to send their input data over the internet, and are trained on unknown data
sources. Can smaller, more targeted models compete? To address this question,
we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive
model trained exclusively on PubMed abstracts and full articles. When
fine-tuned, BioMedLM can produce strong multiple-choice biomedical
question-answering results competitive with much larger models, such as
achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical
Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to
patient questions on medical topics. This demonstrates that smaller models can
potentially serve as transparent, privacy-preserving, economical and
environmentally friendly foundations for particular NLP applications, such as
in biomedicine. The model is available on the Hugging Face Hub:
https://huggingface.co/stanford-crfm/BioMedLM.
6604d1a9596db3db5dc65fac
Elliot Bolton
6604d1a9596db3db5dc65fad
Abhinav Venigalla
6604d1a9596db3db5dc65fae
Michihiro Yasunaga
claimed_verified
2025-02-21T16:39:31.050Z
621e9388345a1d9ab65391c3
/avatars/d009c811dc63224f8eba1889fefad04b.svg
Michihiro Yasunaga
user
michiyasunaga
6604d1a9596db3db5dc65faf
David Hall
6604d1a9596db3db5dc65fb0
Betty Xiong
6604d1a9596db3db5dc65fb1
Tony Lee
6604d1a9596db3db5dc65fb2
Roxana Daneshjou
6604d1a9596db3db5dc65fb3
Jonathan Frankle
6604d1a9596db3db5dc65fb4
Percy Liang
6604d1a9596db3db5dc65fb5
Michael Carbin
6604d1a9596db3db5dc65fb6
Christopher D. Manning
BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance
on a wide variety of biomedical NLP tasks. However, these models have hundreds
of billions of parameters, are computationally expensive to run, require users
to send their input data over the internet, and are trained on unknown data
sources. Can smaller, more targeted models compete? To address this question,
we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive
model trained exclusively on PubMed abstracts and full articles. When
fine-tuned, BioMedLM can produce strong multiple-choice biomedical
question-answering results competitive with much larger models, such as
achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical
Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to
patient questions on medical topics. This demonstrates that smaller models can
potentially serve as transparent, privacy-preserving, economical and
environmentally friendly foundations for particular NLP applications, such as
in biomedicine. The model is available on the Hugging Face Hub:
https://huggingface.co/stanford-crfm/BioMedLM.
BioMedLM, a 2.7 billion parameter GPT-style model trained on PubMed, achieves competitive performance on biomedical NLP tasks with smaller computational requirements.
GPT-style autoregressive model fine-tuning MedMCQA MMLU Medical Genetics