VaMP: Variational Multi-Modal Prompt Learning
Silin Cheng, Kai Han
Vision-language models (VLMs), such as CLIP, have shown strong generalization under zero-shot settings, yet adapting them to downstream tasks with limited supervision remains a significant challenge. Existing multi-modal prompt learning methods typically rely on fixed, shared prompts and deterministic parameters, which limits their ability to capture instance-level variation or model uncertainty across diverse tasks and domains. We propose VaMP, a variational framework for prompt adaptation that enables sample-specific, uncertainty-aware tuning in multi-modal representation learning. VaMP generates instance-conditioned prompts by sampling from a learned posterior distribution, allowing the model to personalize its behavior based on input content. To guide this process with both local and global semantics, we introduce a class-aware prior constructed from the instance representation and class prototype. We formulate prompt tuning as variational inference over latent prompt representations and train the entire framework end-to-end via reparameterized sampling. Experiments on few-shot and domain generalization benchmarks show that VaMP achieves state-of-the-art performance among prompt tuning methods, highlighting the benefits of modeling both uncertainty and task structure in multi-modal prompt adaptation.