An Introduction to Vision-Language Modeling
Following the recent popularity of Large Language Models (LLMs), several
attempts have been made to extend them to the visual domain. From having a
visual assistant that could guide us through unfamiliar environments to
generative models that produce images using only a high-level text description,
the vision-language model (VLM) applications will significantly impact our
relationship with technology. However, there are many challenges that need to
be addressed to improve the reliability of those models. While language is
discrete, vision evolves in a much higher dimensional space in which concepts
cannot always be easily discretized. To better understand the mechanics behind
mapping vision to language, we present this introduction to VLMs which we hope
will help anyone who would like to enter the field. First, we introduce what
VLMs are, how they work, and how to train them. Then, we present and discuss
approaches to evaluate VLMs. Although this work primarily focuses on mapping
images to language, we also discuss extending VLMs to videos.
66554c1f91494de1ddb4d25e
Florian Bordes
extracted_confirmed
2024-05-28T10:51:48.271Z
63166336894404e2506b8811
/avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
Florian Bordes
user
Fbordes
66554c1f91494de1ddb4d25f
Richard Yuanzhe Pang
admin_assigned
2024-05-28T07:23:32.096Z
64d50fa59a6a7ae984d90852
/avatars/26b78c151e707e3f9a9516c877d18ff7.svg
Richard Pang
user
yzpang
66554c1f91494de1ddb4d260
Anurag Ajay
admin_assigned
2024-05-28T07:23:44.241Z
64a7345b23622f7f188cea55
/avatars/069162d6d85bb0d9e810c24a867aff7b.svg
Anurag Ajay
user
anuragajay
66554c1f91494de1ddb4d261
Alexander C. Li
66554c1f91494de1ddb4d262
Adrien Bardes
66554c1f91494de1ddb4d263
Suzanne Petryk
66554c1f91494de1ddb4d264
Oscar Mañas
admin_assigned
2024-05-28T07:24:15.589Z
6355d5c460c1b72f62693983
https://cdn-avatars.huggingface.co/v1/production/uploads/6355d5c460c1b72f62693983/RA4FJCgPyqMXgTaSvBUGq.jpeg
Oscar Mañas
user
oscmansan
66554c1f91494de1ddb4d265
Zhiqiu Lin
admin_assigned
2024-05-28T07:25:06.934Z
64c170190bfb901b04399295
/avatars/c30ce7566ae3497ddc989ec8918d37cc.svg
Zhiqiu Lin
user
zhiqiulin
66554c1f91494de1ddb4d266
Anas Mahmoud
admin_assigned
2024-05-28T07:25:16.518Z
64a588d26c0223510c6047a9
/avatars/6cc422be5cd2f0f78c0b0cc9c3743e02.svg
Anas Mahmoud
user
nasmahmoud
66554c1f91494de1ddb4d267
Bargav Jayaraman
admin_assigned
2024-05-28T07:25:22.476Z
661ddb6c554eb4b8b86f872a
/avatars/6829d1466ea28a363ca7c812401fe332.svg
Bargav Jayaraman
user
bargavj
66554c1f91494de1ddb4d268
Mark Ibrahim
claimed_verified
2024-11-18T15:20:11.701Z
64c17345e82e55936cf971bc
/avatars/b39c5637e00291410620afb7c770d587.svg
Mark Ibrahim
user
marksibrahim
66554c1f91494de1ddb4d269
Melissa Hall
claimed_verified
2024-07-18T09:08:45.585Z
6697bfb4acbced03ebd133f8
/avatars/a7d41367c6f7974303feb5af2a387ce4.svg
Melissa Hall
user
melissah14
66554c1f91494de1ddb4d26a
Yunyang Xiong
claimed_verified
2024-05-28T07:18:54.764Z
65304b62e7535baecd85d080
/avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
Yunyang Xiong
user
yunyangx
66554c1f91494de1ddb4d26b
Jonathan Lebensold
admin_assigned
2024-05-28T07:25:36.765Z
6515a093c33a8b191953902c
/avatars/871ae82d37c63bc5e2dd15726b845d41.svg
Jonathan Lebensold
user
jlebensold
66554c1f91494de1ddb4d26c
Candace Ross
admin_assigned
2024-05-28T07:25:52.567Z
628709d57cc3b7bc1b95fcf0
/avatars/98e0d9695527125a0409d1098cdceaa4.svg
Candace Ross
user
candaceross
66554c1f91494de1ddb4d26d
Srihari Jayakumar
66554c1f91494de1ddb4d26e
Chuan Guo
admin_assigned
2024-05-28T07:26:10.065Z
64c933cfdc59e43a6109b065
/avatars/b7f64fae8df8704c4911065759abfcd3.svg
Chuan Guo
user
chuanguo
66554c1f91494de1ddb4d26f
Diane Bouchacourt
admin_assigned
2024-05-28T07:26:16.384Z
641e01d13bae5a776368311b
/avatars/1e983fb59f475b3252e201f44eec4657.svg
Diane Bouchacourt
user
DianeB
66554c1f91494de1ddb4d270
Haider Al-Tahan
admin_assigned
2024-05-28T07:26:26.010Z
63654c5edaa65e06f90cc5ee
/avatars/11fd0c361531f97b16b91629d0083b92.svg
haider
user
haideraltahan
66554c1f91494de1ddb4d271
Karthik Padthe
admin_assigned
2024-05-28T07:26:32.153Z
64c01530bd4b58768d8b46b5
/avatars/0f3c0cdab16e3078b37569d73ca03792.svg
Karthik padthe
user
Karthikpadthe
66554c1f91494de1ddb4d272
Vasu Sharma
admin_assigned
2024-05-28T07:26:54.171Z
65492730a3d1682f79f3ab7a
/avatars/fc62605bdf10947dab393305c51ace96.svg
Vasu Sharma
user
vasusharma55
66554c1f91494de1ddb4d273
Hu Xu
66554c1f91494de1ddb4d274
Xiaoqing Ellen Tan
admin_assigned
2024-05-28T07:27:42.421Z
6345ab8ffe134dfd7a0d1059
/avatars/159a364ac4f8a2ed001346d4cbb7f2e3.svg
Ellen Tan
user
ellenxtan
66554c1f91494de1ddb4d275
Megan Richards
66554c1f91494de1ddb4d276
Samuel Lavoie
admin_assigned
2024-05-28T07:28:27.013Z
64fdd1509132c7f62a49b16b
/avatars/b3da596b4f622e0acb0c1d0e2ca8643e.svg
Samuel Lavoie
user
lavoiems
66554c1f91494de1ddb4d277
Pietro Astolfi
admin_assigned
2024-05-28T07:28:34.950Z
646388dfd4d34b01f45d87c3
/avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
Pietro Astolfi
user
pietroastolfi
66554c1f91494de1ddb4d278
Reyhane Askari Hemmat
66554c1f91494de1ddb4d279
Jun Chen
66554c1f91494de1ddb4d27a
Kushal Tirumala
admin_assigned
2024-05-28T07:29:57.624Z
6429dac775bcc24c5e579f5c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GI73-WJ5WjdB-KUUNGkdv.jpeg
Kushal Tirumala
user
kushaltirumala
66554c1f91494de1ddb4d27b
Rim Assouel
admin_assigned
2024-05-28T07:30:05.857Z
65f49a8c9a664d06a3d7d10e
/avatars/46a8b6f4f75fe9c375794398cdbc7a42.svg
Rim Assouel
user
rassouel
66554c1f91494de1ddb4d27c
Mazda Moayeri
claimed_verified
2025-06-02T07:58:59.637Z
63449874ee1504dbcd59af3d
/avatars/57a805d82d16de9544c98585bd7a3e55.svg
MazdaM
user
mmoayeri
66554c1f91494de1ddb4d27d
Arjang Talattof
66554c1f91494de1ddb4d27e
Kamalika Chaudhuri
claimed_verified
2024-12-04T09:47:30.438Z
674fd6355d220432321ae8da
/avatars/69839b5cb44a159c711a4dd6d8046477.svg
Kamalika Chaudhuri
user
kamalikac
66554c1f91494de1ddb4d27f
Zechun Liu
admin_assigned
2024-05-28T07:30:27.107Z
660f893bae89429c07a32cdb
/avatars/27442b1dab58114cfe10220c040c1156.svg
Zechun Liu
user
zechunliu
66554c1f91494de1ddb4d280
Xilun Chen
admin_assigned
2024-05-28T07:30:44.156Z
62ffccd18813dd8628172c6c
/avatars/62741be5967a74758a2b7914aa842ec8.svg
Xilun Chen
user
ccsasuke
66554c1f91494de1ddb4d281
Quentin Garrido
admin_assigned
2024-05-28T07:30:51.045Z
63049022412a1b9d381b9dcb
/avatars/cac2a540fda40c5ae558968c50385f46.svg
Quentin Garrido
user
garridoq
66554c1f91494de1ddb4d282
Karen Ullrich
66554c1f91494de1ddb4d283
Aishwarya Agrawal
admin_assigned
2024-05-28T07:31:03.756Z
644fff2d17b6189cda58cb0e
/avatars/03b367b9033c655129e48bd94bd245ee.svg
Aishwarya Agrawal
user
aagrawal
66554c1f91494de1ddb4d284
Kate Saenko
admin_assigned
2024-05-28T07:31:12.358Z
6272e87b72307af646a5372c
/avatars/258f3977ae88d747892dcf83b2c56c2e.svg
Kate Saenko
user
KateSaenko
66554c1f91494de1ddb4d285
Asli Celikyilmaz
66554c1f91494de1ddb4d286
Vikas Chandra
admin_assigned
2024-05-28T07:31:35.359Z
6566681f7b5ed0735812af32
/avatars/424905abb2973954b0850c592743b6fb.svg
Vikas Chandra
user
vchandra
An Introduction to Vision-Language Modeling
Following the recent popularity of Large Language Models (LLMs), several
attempts have been made to extend them to the visual domain. From having a
visual assistant that could guide us through unfamiliar environments to
generative models that produce images using only a high-level text description,
the vision-language model (VLM) applications will significantly impact our
relationship with technology. However, there are many challenges that need to
be addressed to improve the reliability of those models. While language is
discrete, vision evolves in a much higher dimensional space in which concepts
cannot always be easily discretized. To better understand the mechanics behind
mapping vision to language, we present this introduction to VLMs which we hope
will help anyone who would like to enter the field. First, we introduce what
VLMs are, how they work, and how to train them. Then, we present and discuss
approaches to evaluate VLMs. Although this work primarily focuses on mapping
images to language, we also discuss extending VLMs to videos.
Introduction to vision-language models (VLMs) covering their applications, training, evaluation, and extension to videos, addressing challenges in mapping visual data to language.
Large Language Models (LLMs) vision-language models (VLMs) visual assistant generative models image to language vision discrete language high-dimensional space concept discretization video to language