μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
Recent advances in microscopy have enabled the rapid generation of terabytes
of image data in cell biology and biomedical research. Vision-language models
(VLMs) offer a promising solution for large-scale biological image analysis,
enhancing researchers' efficiency, identifying new image biomarkers, and
accelerating hypothesis generation and scientific discovery. However, there is
a lack of standardized, diverse, and large-scale vision-language benchmarks to
evaluate VLMs' perception and cognition capabilities in biological image
understanding. To address this gap, we introduce {\mu}-Bench, an expert-curated
benchmark encompassing 22 biomedical tasks across various scientific
disciplines (biology, pathology), microscopy modalities (electron,
fluorescence, light), scales (subcellular, cellular, tissue), and organisms in
both normal and abnormal states. We evaluate state-of-the-art biomedical,
pathology, and general VLMs on {\mu}-Bench and find that: i) current models
struggle on all categories, even for basic tasks such as distinguishing
microscopy modalities; ii) current specialist models fine-tuned on biomedical
data often perform worse than generalist models; iii) fine-tuning in specific
microscopy domains can cause catastrophic forgetting, eroding prior biomedical
knowledge encoded in their base model. iv) weight interpolation between
fine-tuned and pre-trained models offers one solution to forgetting and
improves general performance across biomedical tasks. We release {\mu}-Bench
under a permissive license to accelerate the research and development of
microscopy foundation models.
6685bfc85a5cdc0baee289cf
Alejandro Lozano
claimed_verified
2024-07-04T08:03:56.249Z
630e56d6f6f6d700f50be105
https://cdn-avatars.huggingface.co/v1/production/uploads/630e56d6f6f6d700f50be105/_gzPXZeL2LP8oalMCriNN.jpeg
Alejandro Lozano
user
Alejandro98
6685bfc85a5cdc0baee289d0
Jeffrey Nirschl
extracted_confirmed
2024-07-03T22:18:57.523Z
6500e67e13f1546526bd373a
/avatars/80ef1ce58e93b773dcbcc2d5b53175c2.svg
Jeff Nirschl
user
jnirschl
6685bfc85a5cdc0baee289d1
James Burgess
extracted_confirmed
2024-07-03T21:23:29.887Z
650871aeb44445e9b3625c7b
https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
James Burgess
user
jmhb
6685bfc85a5cdc0baee289d2
Sanket Rajan Gupte
extracted_confirmed
2024-07-03T22:21:03.466Z
64e80e69ad1bac272b704647
/avatars/540c254d9945efba4ec428241a729a38.svg
Sanket Gupte
user
sanketg
6685bfc85a5cdc0baee289d3
Yuhui Zhang
extracted_confirmed
2024-07-03T22:01:50.861Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
user
yuhuizhang
6685bfc85a5cdc0baee289d4
Alyssa Unell
extracted_pending
2024-07-03T21:17:01.667Z
6623f8d689e599f5169cb56f
/avatars/24003f9ce9a1a7e34f24bf5b0fc2965d.svg
Alyssa Unell
user
aunell
6685bfc85a5cdc0baee289d5
Serena Yeung-Levy
extracted_pending
2025-01-07T02:05:14.400Z
677c8b2e92550a07fcad0f50
/avatars/2be26e8f25e98cfe5b1d227ee0409cd0.svg
Serena Yeung-Levy
user
yeunglevy
μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
Recent advances in microscopy have enabled the rapid generation of terabytes
of image data in cell biology and biomedical research. Vision-language models
(VLMs) offer a promising solution for large-scale biological image analysis,
enhancing researchers' efficiency, identifying new image biomarkers, and
accelerating hypothesis generation and scientific discovery. However, there is
a lack of standardized, diverse, and large-scale vision-language benchmarks to
evaluate VLMs' perception and cognition capabilities in biological image
understanding. To address this gap, we introduce {\mu}-Bench, an expert-curated
benchmark encompassing 22 biomedical tasks across various scientific
disciplines (biology, pathology), microscopy modalities (electron,
fluorescence, light), scales (subcellular, cellular, tissue), and organisms in
both normal and abnormal states. We evaluate state-of-the-art biomedical,
pathology, and general VLMs on {\mu}-Bench and find that: i) current models
struggle on all categories, even for basic tasks such as distinguishing
microscopy modalities; ii) current specialist models fine-tuned on biomedical
data often perform worse than generalist models; iii) fine-tuning in specific
microscopy domains can cause catastrophic forgetting, eroding prior biomedical
knowledge encoded in their base model. iv) weight interpolation between
fine-tuned and pre-trained models offers one solution to forgetting and
improves general performance across biomedical tasks. We release {\mu}-Bench
under a permissive license to accelerate the research and development of
microscopy foundation models.
An expert-curated benchmark evaluates the performance of vision-language models on diverse biomedical tasks, highlighting limitations of current models and proposing weight interpolation as a solution to catastrophic forgetting.
vision-language models biomedical image analysis expert-curated benchmark biomedical tasks microscopy modalities subcellular cellular tissue specialist models generalist models catastrophic forgetting weight interpolation microscopy foundation models