Cached Token Similarity Is a Strong Prior for Fine-grained Visual Question Answering
Liangyu Zhong, Fabio Rosenthal, Joachim Sicking, Fabian Hüger, Thorsten Bagdonat, Hanno Gottschalk, Leo Schwinn
While Multimodal Large Language Models (MLLMs) offer great perception and reasoning capabilities for image-text input, fine-grained Visual Question Answering (VQA) focusing on small details still remains a challenge.   Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations.  We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region.  This is accomplished in four steps: first, we identify the target object(s) in the prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region.  As a result of this informed search strategy, our method achieves strong performance across four fine-grained VQA datasets and two types of MLLM.  It outperforms three existing visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, with 3 - 6.5 x higher efficiency.  Finally, we perform an ablation study to assess the impact of key design choices.We plan to release our code upon acceptance.