OmniFusion Technical Report
Last year, multimodal architectures served up a revolution in AI-based
approaches and solutions, extending the capabilities of large language models
(LLM). We propose an OmniFusion model based on a pretrained LLM and
adapters for visual modality. We evaluated and compared several architecture
design principles for better text and visual data coupling: MLP and transformer
adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
fusing approach, image encoding method (whole image or tiles encoding) and two
7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
visual-language benchmarks show the top score for the best OmniFusion setup in
terms of different VQA tasks in comparison with open-source LLaVA-like
solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
also propose a variety of situations, where OmniFusion provides highly-detailed
answers in different domains: housekeeping, sightseeing, culture, medicine,
handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
model is an open-source solution with weights, training and inference scripts
available at https://github.com/AIRI-Institute/OmniFusion.
6615f762c7f45b3f8930674d
Elizaveta Goncharova
admin_assigned
2024-04-25T07:37:04.576Z
6310ff34bc152fa3e810c186
/avatars/bfd63bcd81548283f5e496e3693bf143.svg
Elizaveta Goncharova
user
Elizaveta
6615f762c7f45b3f8930674e
Anton Razzhigaev
claimed_verified
2024-04-10T10:58:25.360Z
6172aaeec8e66e2aa84c06b9
https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
Anton Razzhigaev
user
razzant
6615f762c7f45b3f8930674f
Matvey Mikhalchuk
claimed_verified
2024-04-11T05:52:36.743Z
64ee45a944f4b3b1bccc02d1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ee45a944f4b3b1bccc02d1/SoidO9HQ4mftzbUPtuBBf.png
Matvey Mikhalchuk
user
matveymih
6615f762c7f45b3f89306750
Maxim Kurkin
claimed_verified
2024-04-10T10:58:16.683Z
634c084c4abe84057588fd63
https://cdn-avatars.huggingface.co/v1/production/uploads/634c084c4abe84057588fd63/-3doVHYwCqbkPQtwVX9aM.jpeg
Maxim Kurkin
user
dondosss
6615f762c7f45b3f89306751
Irina Abdullaeva
claimed_verified
2024-04-11T14:54:46.017Z
648444f14bb88d273c4cab7f
https://cdn-avatars.huggingface.co/v1/production/uploads/648444f14bb88d273c4cab7f/iGjkOf34bcB6K2-J5C_0U.jpeg
Irina Abdullaeva
user
IrinaAbdullaeva
6615f762c7f45b3f89306752
Matvey Skripkin
claimed_verified
2024-04-23T07:15:41.309Z
6626c5d0a329de26e7eb16fa
/avatars/124f389f768fb666efd8b5a9b54c3b3c.svg
Matvey Skripkin
user
barracuda049
6615f762c7f45b3f89306753
Ivan Oseledets
admin_assigned
2024-04-25T07:37:10.732Z
6169a581d05945bfd8718dfa
/avatars/1892ab06a7ddb557232777de3cbec470.svg
Ivan Oseledets
user
oseledets
6615f762c7f45b3f89306754
Denis Dimitrov
claimed_verified
2024-06-28T13:07:37.859Z
6669a678465d1d802181e456
https://cdn-avatars.huggingface.co/v1/production/uploads/6669a678465d1d802181e456/ZCthBBhDFQnh0bBkgUQUU.png
Denis Dimitrov
user
dendimitrov
6615f762c7f45b3f89306755
Andrey Kuznetsov
claimed_verified
2024-04-11T05:52:42.541Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
OmniFusion Technical Report
Last year, multimodal architectures served up a revolution in AI-based
approaches and solutions, extending the capabilities of large language models
(LLM). We propose an OmniFusion model based on a pretrained LLM and
adapters for visual modality. We evaluated and compared several architecture
design principles for better text and visual data coupling: MLP and transformer
adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
fusing approach, image encoding method (whole image or tiles encoding) and two
7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
visual-language benchmarks show the top score for the best OmniFusion setup in
terms of different VQA tasks in comparison with open-source LLaVA-like
solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
also propose a variety of situations, where OmniFusion provides highly-detailed
answers in different domains: housekeeping, sightseeing, culture, medicine,
handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
model is an open-source solution with weights, training and inference scripts
available at https://github.com/AIRI-Institute/OmniFusion.
The OmniFusion model, leveraging pretrained LLMs and visual adapters, demonstrates superior performance across multiple visual-language benchmarks compared to open-source alternatives.
multimodal architectures OmniFusion model large language models (LLM) adapters visual modality MLP transformer adapters CLIP ViT-based encoders SigLIP InternVIT image encoding tile encoding VQA tasks LLaVA-like solutions VizWiz Pope MM-Vet ScienceQA MMBench TextVQA VQAv2 MMMU handcrafted equations recognition Mistral-based OmniFusion