MolmoAct: Action Reasoning Models that can Reason in Space
Reasoning is central to purposeful action, yet most robotic foundation models
map perception and instructions directly to control, which limits adaptability,
generalization, and semantic grounding. We introduce Action Reasoning Models
(ARMs), a class of vision-language-action models that integrate perception,
planning, and control through a structured three-stage pipeline. Our model,
MolmoAct, encodes observations and instructions into depth-aware perception
tokens, generates mid-level spatial plans as editable trajectory traces, and
predicts precise low-level actions, enabling explainable and steerable
behavior. MolmoAct-7B-D achieves strong performance across simulation and
real-world settings: 70.5% zero-shot accuracy on SimplerEnv Visual Matching
tasks, surpassing closed-source Pi-0 and GR00T N1; 86.6% average success on
LIBERO, including an additional 6.3% gain over ThinkAct on long-horizon tasks;
and in real-world fine-tuning, an additional 10% (single-arm) and an additional
22.7% (bimanual) task progression over Pi-0-FAST. It also outperforms baselines
by an additional 23.3% on out-of-distribution generalization and achieves top
human-preference scores for open-ended instruction following and trajectory
steering. Furthermore, we release, for the first time, the MolmoAct Dataset --
a mid-training robot dataset comprising over 10,000 high quality robot
trajectories across diverse scenarios and tasks. Training with this dataset
yields an average 5.5% improvement in general performance over the base model.
We release all model weights, training code, our collected dataset, and our
action reasoning dataset, establishing MolmoAct as both a state-of-the-art
robotics foundation model and an open blueprint for building ARMs that
transform perception into purposeful action through structured reasoning.
Blogpost: https://allenai.org/blog/molmoact
689aa8affab6fdd2e52ac42b
Jason Lee
claimed_verified
2025-08-13T07:21:13.282Z
6473d133bf9b32c6bbcba926
/avatars/c5eae19880dc021c53ec4cf87eeeadee.svg
Jason Lee
user
jaslee20
689aa8affab6fdd2e52ac42c
Jiafei Duan
689aa8affab6fdd2e52ac42d
Haoquan Fang
claimed_verified
2025-08-13T07:21:09.620Z
6402a262cad36eb2b0fdd14b
https://cdn-avatars.huggingface.co/v1/production/uploads/6402a262cad36eb2b0fdd14b/Er0k6QnQFUpBeExTjOMBr.jpeg
Haoquan Fang
user
hqfang
689aa8affab6fdd2e52ac42e
Yuquan Deng
689aa8affab6fdd2e52ac42f
Shuo Liu
689aa8affab6fdd2e52ac430
Boyang Li
689aa8affab6fdd2e52ac431
Bohan Fang
689aa8affab6fdd2e52ac432
Jieyu Zhang
689aa8affab6fdd2e52ac433
Yi Ru Wang
689aa8affab6fdd2e52ac434
Sangho Lee
689aa8affab6fdd2e52ac435
Winson Han
689aa8affab6fdd2e52ac436
Wilbert Pumacay
689aa8affab6fdd2e52ac437
Angelica Wu
689aa8affab6fdd2e52ac438
Rose Hendrix
689aa8affab6fdd2e52ac439
Karen Farley
689aa8affab6fdd2e52ac43a
Eli VanderBilt
689aa8affab6fdd2e52ac43b
Ali Farhadi
689aa8affab6fdd2e52ac43c
Dieter Fox
689aa8affab6fdd2e52ac43d
Ranjay Krishna
MolmoAct: Action Reasoning Models that can Reason in Space
Reasoning is central to purposeful action, yet most robotic foundation models
map perception and instructions directly to control, which limits adaptability,
generalization, and semantic grounding. We introduce Action Reasoning Models
(ARMs), a class of vision-language-action models that integrate perception,
planning, and control through a structured three-stage pipeline. Our model,
MolmoAct, encodes observations and instructions into depth-aware perception
tokens, generates mid-level spatial plans as editable trajectory traces, and
predicts precise low-level actions, enabling explainable and steerable
behavior. MolmoAct-7B-D achieves strong performance across simulation and
real-world settings: 70.5% zero-shot accuracy on SimplerEnv Visual Matching
tasks, surpassing closed-source Pi-0 and GR00T N1; 86.6% average success on
LIBERO, including an additional 6.3% gain over ThinkAct on long-horizon tasks;
and in real-world fine-tuning, an additional 10% (single-arm) and an additional
22.7% (bimanual) task progression over Pi-0-FAST. It also outperforms baselines
by an additional 23.3% on out-of-distribution generalization and achieves top
human-preference scores for open-ended instruction following and trajectory
steering. Furthermore, we release, for the first time, the MolmoAct Dataset --
a mid-training robot dataset comprising over 10,000 high quality robot
trajectories across diverse scenarios and tasks. Training with this dataset
yields an average 5.5% improvement in general performance over the base model.
We release all model weights, training code, our collected dataset, and our
action reasoning dataset, establishing MolmoAct as both a state-of-the-art
robotics foundation model and an open blueprint for building ARMs that
transform perception into purposeful action through structured reasoning.
Blogpost: https://allenai.org/blog/molmoact
Action Reasoning Models (ARMs) integrate perception, planning, and control to enable adaptable and explainable robotic behavior, achieving superior performance across various tasks and settings.
Action Reasoning Models ARMs MolmoAct depth-aware perception tokens mid-level spatial plans trajectory traces low-level actions SimplerEnv Visual Matching LIBERO ThinkAct MolmoAct Dataset open-ended instruction following trajectory steering