DiffCLIP: Differential Attention Meets CLIP
We propose DiffCLIP, a novel vision-language model that extends the
differential attention mechanism to CLIP architectures. Differential attention
was originally developed for large language models to amplify relevant context
while canceling out noisy information. In this work, we integrate this
mechanism into CLIP's dual encoder (image and text) framework. With minimal
additional parameters, DiffCLIP achieves superior performance on image-text
understanding tasks. Across zero-shot classification, retrieval, and robustness
benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably,
these gains come with negligible computational overhead, demonstrating that
differential attention can significantly enhance multi-modal representations
without sacrificing efficiency. Code can be found at
https://github.com/hammoudhasan/DiffCLIP.
67d02a6da2b648cac0c37359
Hasan Abed Al Kader Hammoud
admin_assigned
2025-03-11T14:30:32.989Z
642b51385bf2355d02a23d15
/avatars/87985347643b2647555f2453fa4d94fb.svg
Hasan Abed Al Kader Hammoud
user
hammh0a
67d02a6da2b648cac0c3735a
Bernard Ghanem
DiffCLIP: Differential Attention Meets CLIP
We propose DiffCLIP, a novel vision-language model that extends the
differential attention mechanism to CLIP architectures. Differential attention
was originally developed for large language models to amplify relevant context
while canceling out noisy information. In this work, we integrate this
mechanism into CLIP's dual encoder (image and text) framework. With minimal
additional parameters, DiffCLIP achieves superior performance on image-text
understanding tasks. Across zero-shot classification, retrieval, and robustness
benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably,
these gains come with negligible computational overhead, demonstrating that
differential attention can significantly enhance multi-modal representations
without sacrificing efficiency. Code can be found at
https://github.com/hammoudhasan/DiffCLIP.
DiffCLIP enhances CLIP's performance on image-text tasks by integrating differential attention, improving accuracy with minimal additional parameters and computational cost.
differential attention mechanism CLIP architectures dual encoder zero-shot classification retrieval robustness benchmarks multi-modal representations