SMMILE: An expert-driven benchmark for multimodal medical in-context learning
Melanie Rieff, Maya Varma, Ossian Rabow, Subathra Adithan, Julie Kim, Ken Chang, Hannah Lee, Nidhi Rohatgi, Christian Bluethgen, Mohamed Muneer, Jean-Benoit Delbrouck, Michael Moor
Multimodal in-context learning (ICL) remains underexplored despite the profound potential it could have in complex application domains such as medicine. Clinicians routinely face a long tail of tasks which they need to learn to solve from few examples, such as considering few relevant previous cases or few differential diagnoses. While multimodal large language models (MLLMs)  have shown impressive advances in medical visual question answering (VQA) or multi-turn chatting, their ability to learn multimodal tasks from context is largely unknown. Here, we introduce Stanford Multimodal Medical In-context Learning benchmark (SMMILE), the first multimodal medical ICL benchmark. A team of 11 clinical experts created ICL problems to scrutinize MLLM’s ability to learn multimodal tasks at inference time from context. We benchmark a suite of 14 diverse vision language models (VLMs) on SMMILE. Our results show that most VLMs gain little from multimodal in-context learning in medicine. Only two of the fourteen evaluated model, GPT-4o and Qwen 2.5-VL-32B, record double-digit improvements (>10 percentage points) over their zero-shot accuracy, and the mean gain across all models is about 4 percentage points. Further analysis reveals a clear recency bias: placing the most relevant example last boosts accuracy by more than 10 points, whereas adding noisy examples can cut performance by up to 6 points. SMMILE thus highlights critical limitations and biases in current medical multimodal AI.