Lumos : Empowering Multimodal LLMs with Scene Text Recognition
We introduce Lumos, the first end-to-end multimodal question-answering system
with text understanding capabilities. At the core of Lumos is a Scene Text
Recognition (STR) component that extracts text from first person point-of-view
images, the output of which is used to augment input to a Multimodal Large
Language Model (MM-LLM). While building Lumos, we encountered numerous
challenges related to STR quality, overall latency, and model inference. In
this paper, we delve into those challenges, and discuss the system
architecture, design choices, and modeling techniques employed to overcome
these obstacles. We also provide a comprehensive evaluation for each component,
showcasing high quality and efficiency.
65cc2ce019683f9817234084
Ashish Shenoy
claimed_verified
2024-02-15T08:28:04.992Z
65cce8307b7a844001bf9fff
/avatars/130d7736cfdd00af1ba9897b79ee7c82.svg
Ashish Shenoy
user
ashishvshenoy
65cc2ce019683f9817234085
Yichao Lu
extracted_pending
2025-03-24T01:34:45.171Z
62d18f063ca54f8303f4dcc8
/avatars/042eb71248350603f1e58ea0a4790345.svg
bgjd
user
dspg
65cc2ce019683f9817234086
Srihari Jayakumar
65cc2ce019683f9817234087
Debojeet Chatterjee
admin_assigned
2024-02-14T12:55:55.009Z
6261c8376c57f74478073c3a
/avatars/8a64178dd148379b298b0bced79295e8.svg
Debojeet Chatterjee
user
debowin
65cc2ce019683f9817234088
Mohsen Moslehpour
65cc2ce019683f9817234089
Pierce Chuang
admin_assigned
2024-02-14T12:56:04.997Z
640bb0a619f883e24a4a0f1e
/avatars/c884d5ed8009dffb3f9414471a8817e3.svg
Pierce Chuang
user
pichuang
65cc2ce019683f981723408a
Abhay Harpale
65cc2ce019683f981723408b
Vikas Bhardwaj
extracted_confirmed
2024-02-14T16:06:55.944Z
628830d687a393458fc33a1c
/avatars/fc8786ceef9188efd19f7ae5b9e30554.svg
Vikas Bhardwaj
user
vikasbhardwaj
65cc2ce019683f981723408c
Di Xu
admin_assigned
2024-02-14T12:56:28.096Z
638194c94761ddfa0091a981
/avatars/ca9974ac8f3e4bdd2806f65e64b383f8.svg
Di Xu
user
dixu
65cc2ce019683f981723408d
Shicong Zhao
65cc2ce019683f981723408e
Longfang Zhao
65cc2ce019683f981723408f
Ankit Ramchandani
65cc2ce019683f9817234090
Xin Luna Dong
65cc2ce019683f9817234091
Anuj Kumar
Lumos : Empowering Multimodal LLMs with Scene Text Recognition
We introduce Lumos, the first end-to-end multimodal question-answering system
with text understanding capabilities. At the core of Lumos is a Scene Text
Recognition (STR) component that extracts text from first person point-of-view
images, the output of which is used to augment input to a Multimodal Large
Language Model (MM-LLM). While building Lumos, we encountered numerous
challenges related to STR quality, overall latency, and model inference. In
this paper, we delve into those challenges, and discuss the system
architecture, design choices, and modeling techniques employed to overcome
these obstacles. We also provide a comprehensive evaluation for each component,
showcasing high quality and efficiency.
Lumos is an end-to-end multimodal question-answering system that integrates Scene Text Recognition and a Multimodal Large Language Model to improve text understanding and efficiency from first-person images.
Scene Text Recognition STR Multimodal Large Language Model MM-LLM first person point-of-view multimodal question-answering