Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
  Multimodal Models
Today's most advanced multimodal models remain proprietary. The strongest
open-weight models rely heavily on synthetic data from proprietary VLMs to
achieve good performance, effectively distilling these closed models into open
ones. As a result, the community is still missing foundational knowledge about
how to build performant VLMs from scratch. We present Molmo, a new family of
VLMs that are state-of-the-art in their class of openness. Our key innovation
is a novel, highly detailed image caption dataset collected entirely from human
annotators using speech-based descriptions. To enable a wide array of user
interactions, we also introduce a diverse dataset mixture for fine-tuning that
includes in-the-wild Q&A and innovative 2D pointing data. The success of our
approach relies on careful choices for the model architecture details, a
well-tuned training pipeline, and, most critically, the quality of our newly
collected datasets, all of which will be released. The best-in-class 72B model
within the Molmo family not only outperforms others in the class of open weight
and data models but also compares favorably against proprietary systems like
GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
evaluation.
  We will be releasing all of our model weights, captioning and fine-tuning
data, and source code in the near future. Select model weights, inference code,
and demo are available at https://molmo.allenai.org.
66f4c5c9b26f10a31416f516
Matt Deitke
claimed_verified
2024-09-26T08:23:04.074Z
61c388aa727d1257bf3cf58b
https://cdn-avatars.huggingface.co/v1/production/uploads/1670871898141-61c388aa727d1257bf3cf58b.jpeg
Matt Deitke
user
mattdeitke
66f4c5c9b26f10a31416f517
Christopher Clark
claimed_verified
2024-10-04T20:11:33.431Z
65c92082f3764ed670119351
/avatars/85ebdd196215175d07397ad2c15a2a9b.svg
Christopher Clark
user
chrisc36
66f4c5c9b26f10a31416f518
Sangho Lee
claimed_verified
2024-12-04T09:47:08.524Z
664fc02d6f16bbd9a168654d
/avatars/9f726ede9973706bc198845464238452.svg
Sangho Lee
user
sanghol
66f4c5c9b26f10a31416f519
Rohun Tripathi
admin_assigned
2024-09-26T09:19:34.767Z
646e5aa2cb6ea6e6b6df2a5f
/avatars/65c63a138d793a17fc317fccc2be53b2.svg
Rohun Tripathi
user
roTripathi
66f4c5c9b26f10a31416f51a
Yue Yang
claimed_verified
2024-09-26T12:49:36.119Z
62f6c68904e5e02f82b04690
https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
Yue Yang
user
yyupenn
66f4c5c9b26f10a31416f51b
Jae Sung Park
admin_assigned
2024-09-26T09:20:14.540Z
6345a11743f4f2d2ed1057ca
/avatars/cba8b194b2517da76021d8cc1814a542.svg
Jae Sung Park
user
jamepark3922
66f4c5c9b26f10a31416f51c
Mohammadreza Salehi
admin_assigned
2024-09-26T09:20:26.387Z
66990ec3aa500cd99d640a19
/avatars/3520cab2b946158b2cdf6ed84f9516e0.svg
Mohammadreza Salehi
user
SMSD75
66f4c5c9b26f10a31416f51d
Niklas Muennighoff
admin_assigned
2024-09-26T09:20:32.338Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
66f4c5c9b26f10a31416f51e
Kyle Lo
66f4c5c9b26f10a31416f51f
Luca Soldaini
admin_assigned
2024-09-26T09:20:44.007Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
66f4c5c9b26f10a31416f520
Jiasen Lu
admin_assigned
2024-09-26T09:20:50.472Z
62b6b0397523238923221df9
/avatars/77068771dd51df7519516cd502a88789.svg
Jiasenlu
user
Jiasenlu
66f4c5c9b26f10a31416f521
Taira Anderson
admin_assigned
2024-09-26T09:20:56.203Z
65de20ad4e73a7dea7fb4f08
/avatars/f3b0ad6cc9417e8ea3f0607fa62824d1.svg
Taira Anderson
user
tairaa
66f4c5c9b26f10a31416f522
Erin Bransom
admin_assigned
2024-09-26T09:21:03.691Z
65bd91cfb54ab5b37dd1e309
/avatars/33a26756df8275b94e8e4f1f8a6e8cbc.svg
Erin Bransom
user
erinbransom
66f4c5c9b26f10a31416f523
Kiana Ehsani
66f4c5c9b26f10a31416f524
Huong Ngo
admin_assigned
2024-09-26T09:21:27.740Z
66e89c8525e09fb294c6ad3b
/avatars/7044e2696487f20b2fe50967d805c43b.svg
Huong Ngo
user
huongn
66f4c5c9b26f10a31416f525
YenSung Chen
admin_assigned
2024-09-26T09:21:37.323Z
6697093a37d2483826562c24
/avatars/0e526b4be6db07e2485f7ef862080339.svg
Chen
user
Yensung
66f4c5c9b26f10a31416f526
Ajay Patel
claimed_verified
2024-10-09T07:38:27.736Z
61c40eeb727d1257bf3cf5ba
https://cdn-avatars.huggingface.co/v1/production/uploads/61c40eeb727d1257bf3cf5ba/hVNbcFjsvwWqWarcGTOdI.jpeg
Ajay Patel
user
AjayP13
66f4c5c9b26f10a31416f527
Mark Yatskar
admin_assigned
2024-09-26T09:22:06.633Z
631f42e1b6628770f6efd87a
/avatars/0fa93b3513ebca737cce26dfa5611cf1.svg
Mark Yatskar
user
myatskar
66f4c5c9b26f10a31416f528
Chris Callison-Burch
admin_assigned
2024-09-26T09:22:13.265Z
6303ce25fc783bfc744216af
/avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
Chris Callison-Burch
user
CCB
66f4c5c9b26f10a31416f529
Andrew Head
admin_assigned
2024-09-26T09:22:19.015Z
6360289588e41d249ecd3e26
/avatars/0e84dbd72f07e99967a8d25cda938efe.svg
Andrew Head
user
ChittyChins
66f4c5c9b26f10a31416f52a
Rose Hendrix
admin_assigned
2024-09-26T09:22:24.427Z
627598f47128d87e9f92f487
/avatars/1f9ae7236b25f573f62e298436c7422a.svg
Rose Hendrix
user
roseh
66f4c5c9b26f10a31416f52b
Favyen Bastani
admin_assigned
2024-09-26T09:22:29.774Z
64dbe43c70446182be8727a8
/avatars/4390ef620010911901ebb69d944e6b98.svg
Favyen Bastani
user
favyen
66f4c5c9b26f10a31416f52c
Eli VanderBilt
66f4c5c9b26f10a31416f52d
Nathan Lambert
claimed_verified
2025-01-06T08:01:02.562Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
66f4c5c9b26f10a31416f52e
Yvonne Chou
admin_assigned
2024-09-26T09:28:48.664Z
647a38bad547ef74714ce526
/avatars/070c3718078c35046b2fc09ed16ae8f2.svg
Yvonne Chou
user
ychou-ai2
66f4c5c9b26f10a31416f52f
Arnavi Chheda
66f4c5c9b26f10a31416f530
Jenna Sparks
admin_assigned
2024-09-26T09:29:31.683Z
65cd45626f58553a7447b342
/avatars/7dacfd296b99822f13a61a09c06aca1c.svg
Jenna Sparks
user
jennasparks
66f4c5c9b26f10a31416f531
Sam Skjonsberg
admin_assigned
2024-09-26T09:29:38.399Z
603572ed7d20da1ddcb2f59b
https://cdn-avatars.huggingface.co/v1/production/uploads/603572ed7d20da1ddcb2f59b/f4suP0L1vzQsJ0OV7WDrS.png
Sam Skjonsberg
user
codeviking
66f4c5c9b26f10a31416f532
Michael Schmitz
66f4c5c9b26f10a31416f533
Aaron Sarnat
admin_assigned
2024-09-26T09:26:25.270Z
65b1520bf7638a13a641a620
https://cdn-avatars.huggingface.co/v1/production/uploads/65b1520bf7638a13a641a620/KTauZL0kXlmYnbkI2lFBG.png
Aaron Sarnat
user
aaronsarnat
66f4c5c9b26f10a31416f534
Byron Bischoff
claimed_verified
2025-04-11T07:26:19.505Z
67f85d757f56304dc72b4961
/avatars/6ae806d25632b250220145feae200058.svg
Byron Bischoff
user
byronbischoff
66f4c5c9b26f10a31416f535
Pete Walsh
66f4c5c9b26f10a31416f536
Chris Newell
admin_assigned
2024-09-26T09:25:39.106Z
669fe6013db54f36c2b1198b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KY10scX-CtLoDF9oITKeY.jpeg
Chris Newell
user
newellce
66f4c5c9b26f10a31416f537
Piper Wolters
admin_assigned
2024-09-26T09:25:25.912Z
6509e8202e4bbde418147207
/avatars/b8926cb9f5f1a8cff36432ae18ee3c32.svg
Piper Wolters
user
piperw
66f4c5c9b26f10a31416f538
Tanmay Gupta
claimed_verified
2025-06-10T09:32:17.622Z
62a108242e30aaf94ec283bb
/avatars/4da19ae99df6a3fa2d81336eb59cdaa7.svg
Tanmay Gupta
user
tanmayg
66f4c5c9b26f10a31416f539
Kuo-Hao Zeng
66f4c5c9b26f10a31416f53a
Jon Borchardt
66f4c5c9b26f10a31416f53b
Dirk Groeneveld
admin_assigned
2024-09-26T09:24:41.590Z
60369745413a78f892e7339c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
Dirk Groeneveld
user
dirkgr
66f4c5c9b26f10a31416f53c
Jen Dumas
66f4c5c9b26f10a31416f53d
Crystal Nam
admin_assigned
2024-09-26T09:24:19.946Z
64a7264f44387a55046f92e5
/avatars/7f6e076a2a535957bc9c7ac4aeae716d.svg
Crystal Nam
user
CrystalN
66f4c5c9b26f10a31416f53e
Sophie Lebrecht
admin_assigned
2024-09-26T09:24:12.561Z
65b301f04c9e50e74a893954
/avatars/f52366959f9e7613576603c0272ff2c5.svg
Sophie Lebrecht
user
Lebrechts
66f4c5c9b26f10a31416f53f
Caitlin Wittlif
admin_assigned
2024-09-26T09:24:05.433Z
652db071b62cf1f8463221e2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-GDOf0IQdomYEEjFD1cMz.png
Caitlin Wittlif
user
caitwallenai
66f4c5c9b26f10a31416f540
Carissa Schoenick
admin_assigned
2024-09-26T09:23:58.908Z
6024546dc1f3c79f98e4b384
https://cdn-avatars.huggingface.co/v1/production/uploads/1612993792778-6024546dc1f3c79f98e4b384.jpeg
Carissa Schoenick
user
CarissaS
66f4c5c9b26f10a31416f541
Oscar Michel
admin_assigned
2024-09-26T09:23:37.182Z
633e387b309a99325093c6ea
https://cdn-avatars.huggingface.co/v1/production/uploads/1665022070673-noauth.jpeg
Oscar  Michel
user
breadroll
66f4c5c9b26f10a31416f542
Ranjay Krishna
admin_assigned
2024-09-26T09:23:29.387Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
66f4c5c9b26f10a31416f543
Luca Weihs
admin_assigned
2024-09-26T09:23:19.856Z
620acbdb3c0931626a7c9297
/avatars/f63b1d225ed81e223d3e8876a5c708c4.svg
Luca Weihs
user
lucaweihs
66f4c5c9b26f10a31416f544
Noah A. Smith
66f4c5c9b26f10a31416f545
Hannaneh Hajishirzi
66f4c5c9b26f10a31416f546
Ross Girshick
admin_assigned
2024-09-26T09:23:04.745Z
645a98a777d040cfd6500e14
/avatars/9695beb63f88952064936c7345d09ee2.svg
Ross Girshick
user
inkynumbers
66f4c5c9b26f10a31416f547
Ali Farhadi
admin_assigned
2024-09-26T09:22:58.044Z
6660d4c1818c5c5ca0f31266
/avatars/1d2972894cb3b9df1900fdb162d9c364.svg
alifarhadi 
user
alifarhadi051
66f4c5c9b26f10a31416f548
Aniruddha Kembhavi
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
  Multimodal Models
Today's most advanced multimodal models remain proprietary. The strongest
open-weight models rely heavily on synthetic data from proprietary VLMs to
achieve good performance, effectively distilling these closed models into open
ones. As a result, the community is still missing foundational knowledge about
how to build performant VLMs from scratch. We present Molmo, a new family of
VLMs that are state-of-the-art in their class of openness. Our key innovation
is a novel, highly detailed image caption dataset collected entirely from human
annotators using speech-based descriptions. To enable a wide array of user
interactions, we also introduce a diverse dataset mixture for fine-tuning that
includes in-the-wild Q&A and innovative 2D pointing data. The success of our
approach relies on careful choices for the model architecture details, a
well-tuned training pipeline, and, most critically, the quality of our newly
collected datasets, all of which will be released. The best-in-class 72B model
within the Molmo family not only outperforms others in the class of open weight
and data models but also compares favorably against proprietary systems like
GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
evaluation.
  We will be releasing all of our model weights, captioning and fine-tuning
data, and source code in the near future. Select model weights, inference code,
and demo are available at https://molmo.allenai.org.
Molmo, a new family of open-weight VLMs, excels through a high-quality human-annotated image caption dataset and a diverse fine-tuning dataset, outperforming both proprietary and open models in benchmark and human evaluations.
multimodal models open-weight models VLMs image caption dataset human annotators speech-based descriptions 2D pointing data model architecture training pipeline GPT-4o Claude 3.5 Gemini 1.5