Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models
Xu Yang, Chen Liu, Ying Wei
Selecting a compact set of visual instruction–following data has emerged as an effective way to align large multimodal models with human intentions while avoiding the cost of full-dataset training. Yet we observe that both full-data and state-of-the-art-method selected-data instruction tuning can inherit underlying dataset biases, leading to biased model behavior due to position bias and spurious correlations. We introduce ARDS, a robustness-aware targeted visual instruction-selection framework that explicitly targets these weaknesses, sidestepping the need for access to downstream data or time-consuming gradient computation. Specifically, we first generate the worst-case evaluation subgroups through visual and textual task-specific perturbations. The robust training mixture is then constructed by prioritizing the samples that are semantically closer to these subgroups in the rich multimodal embedding space. Extensive experiments show that ARDS substantially boosts both robustness and data efficiency for visual instruction tuning.  We also showcase that the robust mixtures produced with a smaller model transfer effectively to larger architectures.