MARVIS: Modality Adaptive Reasoning over VISualizations
Scientific applications of machine learning often rely on small, specialized
models tuned to particular domains. Such models often achieve excellent
performance, but lack flexibility. Foundation models offer versatility, but
typically underperform specialized approaches, especially on non-traditional
modalities and long-tail domains. We propose MARVIS (Modality Adaptive
Reasoning over VISualizations), a training-free method that enables even small
vision-language models to predict any data modality with high accuracy. MARVIS
transforms latent embedding spaces into visual representations and then
leverages the spatial and fine-grained reasoning skills of VLMs to successfully
interpret and utilize them. MARVIS achieves competitive performance on vision,
audio, biological, and tabular domains using a single 3B parameter model,
achieving results that beat Gemini by 16\% on average and approach specialized
methods, without exposing personally identifiable information (P.I.I.) or
requiring any domain-specific training. We open source our code and datasets at
https://github.com/penfever/marvis
6865e0688c83dab5f72d1e33
Benjamin Feuer
claimed_verified
2025-07-04T07:49:59.026Z
62f7f4efe7c1c9bf10c81465
https://cdn-avatars.huggingface.co/v1/production/uploads/62f7f4efe7c1c9bf10c81465/AYlOg0fkP1o4GAP-8Y3xt.jpeg
Benjamin Feuer
user
penfever
6865e0688c83dab5f72d1e34
Lennart Purucker
claimed_verified
2025-07-03T16:19:04.420Z
677d00ac70b9142c01cc90f9
https://cdn-avatars.huggingface.co/v1/production/uploads/677d00ac70b9142c01cc90f9/c2utzkEFwdUHPLN8-_AHb.png
Lennart Purucker
user
LennartPurucker
6865e0688c83dab5f72d1e35
Oussama Elachqar
6865e0688c83dab5f72d1e36
Chinmay Hegde
MARVIS: Modality Adaptive Reasoning over VISualizations
Scientific applications of machine learning often rely on small, specialized
models tuned to particular domains. Such models often achieve excellent
performance, but lack flexibility. Foundation models offer versatility, but
typically underperform specialized approaches, especially on non-traditional
modalities and long-tail domains. We propose MARVIS (Modality Adaptive
Reasoning over VISualizations), a training-free method that enables even small
vision-language models to predict any data modality with high accuracy. MARVIS
transforms latent embedding spaces into visual representations and then
leverages the spatial and fine-grained reasoning skills of VLMs to successfully
interpret and utilize them. MARVIS achieves competitive performance on vision,
audio, biological, and tabular domains using a single 3B parameter model,
achieving results that beat Gemini by 16\% on average and approach specialized
methods, without exposing personally identifiable information (P.I.I.) or
requiring any domain-specific training. We open source our code and datasets at
https://github.com/penfever/marvis
MARVIS, a training-free method, enhances small vision-language models to predict across various data modalities with high accuracy using latent embeddings and spatial reasoning.
vision-language models latent embedding spaces spatial reasoning vision audio biological tabular domains MARVIS Gemini personally identifiable information (P.I.I.)