No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language
  Models
Embedding vision-language models (VLMs) are typically pretrained with short
text windows (<77 tokens), which forces the truncation of long-format captions.
Yet, the distribution of biomedical captions from large-scale open source
literature reveals that a huge portion of captions far exceed 77 tokens. To
this end, we investigate the impact of pretraining on long-format biomedical
captions by extending the context length of text encoders in VLMs. We find that
longer context (thus, enabling additional supervision provided in long-format
captions) correlates with better retrieval and classification performance.
Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M
image-caption pairs enriched with context-aware descriptions from full-text
articles, providing longer and additional textual supervision. Using
BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a
text encoder supporting windows of up to 512 tokens. Our model extends context
capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption
retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in
Recall@1 and +2% average improvements in classification, while also converging
faster than short-context. Our results demonstrate that long-context modeling
is a promising direction for advancing biomedical VLMs.
68e5eb39975ac4c405ef215a
Min Woo Sun
claimed_verified
2025-10-09T06:51:42.299Z
65ac61120844d9e0d67a9f89
https://cdn-avatars.huggingface.co/v1/production/uploads/65ac61120844d9e0d67a9f89/wvXAMRs44oww2M57ZJGqT.jpeg
Min Woo Sun
user
minwoosun
68e5eb39975ac4c405ef215b
Alejandro Lozano
68e5eb39975ac4c405ef215c
Javier Gamazo Tejero
68e5eb39975ac4c405ef215d
Vishwesh Nath
68e5eb39975ac4c405ef215e
Xiao Xiao Sun
68e5eb39975ac4c405ef215f
James Burgess
68e5eb39975ac4c405ef2160
Yuhui Zhang
68e5eb39975ac4c405ef2161
Kun Yuan
68e5eb39975ac4c405ef2162
Robert Tibshirani
68e5eb39975ac4c405ef2163
Sean Huver
68e5eb39975ac4c405ef2164
Serena Yeung-Levy
No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language
  Models
Embedding vision-language models (VLMs) are typically pretrained with short
text windows (<77 tokens), which forces the truncation of long-format captions.
Yet, the distribution of biomedical captions from large-scale open source
literature reveals that a huge portion of captions far exceed 77 tokens. To
this end, we investigate the impact of pretraining on long-format biomedical
captions by extending the context length of text encoders in VLMs. We find that
longer context (thus, enabling additional supervision provided in long-format
captions) correlates with better retrieval and classification performance.
Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M
image-caption pairs enriched with context-aware descriptions from full-text
articles, providing longer and additional textual supervision. Using
BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a
text encoder supporting windows of up to 512 tokens. Our model extends context
capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption
retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in
Recall@1 and +2% average improvements in classification, while also converging
faster than short-context. Our results demonstrate that long-context modeling
is a promising direction for advancing biomedical VLMs.
Extending the context length of text encoders in vision-language models improves performance on biomedical caption tasks by utilizing longer and more detailed descriptions.
embedding vision-language models VLMs text encoders biomedical captions context length token windows long-format captions BIOMEDICA-LongCAP BMC-LongCLIP long-context modeling Recall@1 classification performance