Towards Language Models That Can See: Computer Vision Through the LENS
  of Natural Language
We propose LENS, a modular approach for tackling computer vision problems by
leveraging the power of large language models (LLMs). Our system uses a
language model to reason over outputs from a set of independent and highly
descriptive vision modules that provide exhaustive information about an image.
We evaluate the approach on pure computer vision settings such as zero- and
few-shot object recognition, as well as on vision and language problems. LENS
can be applied to any off-the-shelf LLM and we find that the LLMs with LENS
perform highly competitively with much bigger and much more sophisticated
systems, without any multimodal training whatsoever. We open-source our code at
https://github.com/ContextualAI/lens and provide an interactive demo.
649ce6f33e5ad160389de1a5
William Berrios
admin_assigned
2023-06-29T11:17:07.044Z
62503f03815d0fd28f847c19
https://cdn-avatars.huggingface.co/v1/production/uploads/1656275265222-62503f03815d0fd28f847c19.jpeg
William Berrios
user
will33am
649ce6f33e5ad160389de1a6
Gautam Mittal
admin_assigned
2023-06-29T11:17:26.887Z
6311b43205cc08a1408d5a78
https://cdn-avatars.huggingface.co/v1/production/uploads/6311b43205cc08a1408d5a78/PMGdqvQ9H8LWnPc3L-KPp.jpeg
Gautam Mittal
user
gbm
649ce6f33e5ad160389de1a7
Tristan Thrush
admin_assigned
2023-06-29T11:17:48.272Z
61e9e3d4e2a95338e04c9f33
https://cdn-avatars.huggingface.co/v1/production/uploads/61e9e3d4e2a95338e04c9f33/c3Pfr2LrD5Dbf0eFkScP6.png
Tristan Thrush
user
Tristan
649ce6f33e5ad160389de1a8
Douwe Kiela
admin_assigned
2023-06-29T11:18:08.050Z
61dc997715b47073db1620dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
Douwe Kiela
user
douwekiela
649ce6f33e5ad160389de1a9
Amanpreet Singh
claimed_verified
2024-02-19T09:36:07.606Z
6230c6ecfd8b720a5648f6c4
https://cdn-avatars.huggingface.co/v1/production/uploads/1652253065548-6230c6ecfd8b720a5648f6c4.jpeg
Amanpreet Singh
user
aps
Towards Language Models That Can See: Computer Vision Through the LENS
  of Natural Language
We propose LENS, a modular approach for tackling computer vision problems by
leveraging the power of large language models (LLMs). Our system uses a
language model to reason over outputs from a set of independent and highly
descriptive vision modules that provide exhaustive information about an image.
We evaluate the approach on pure computer vision settings such as zero- and
few-shot object recognition, as well as on vision and language problems. LENS
can be applied to any off-the-shelf LLM and we find that the LLMs with LENS
perform highly competitively with much bigger and much more sophisticated
systems, without any multimodal training whatsoever. We open-source our code at
https://github.com/ContextualAI/lens and provide an interactive demo.
LENS uses language models to reason over outputs from vision modules, achieving competitive performance in vision and vision-language tasks without multimodal training.
LENS large language models (LLMs) vision modules zero-shot object recognition few-shot object recognition multimodal training