Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
Physical AI systems need to perceive, understand, and perform complex actions
in the physical world. In this paper, we present the Cosmos-Reason1 models that
can understand the physical world and generate appropriate embodied decisions
(e.g., next step action) in natural language through long chain-of-thought
reasoning processes. We begin by defining key capabilities for Physical AI
reasoning, with a focus on physical common sense and embodied reasoning. To
represent physical common sense, we use a hierarchical ontology that captures
fundamental knowledge about space, time, and physics. For embodied reasoning,
we rely on a two-dimensional ontology that generalizes across different
physical embodiments. Building on these capabilities, we develop two multimodal
large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B. We curate data
and train our models in four stages: vision pre-training, general supervised
fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
as the post-training. To evaluate our models, we build comprehensive benchmarks
for physical common sense and embodied reasoning according to our ontologies.
Evaluation results show that Physical AI SFT and reinforcement learning bring
significant improvements. To facilitate the development of Physical AI, we will
make our code and pre-trained models available under the NVIDIA Open Model
License at https://github.com/nvidia-cosmos/cosmos-reason1.
67dcadafb2cd7d4f3a266037
NVIDIA
67dcadafb2cd7d4f3a266039
Alisson Azzolini
67dcadafb2cd7d4f3a26603a
Hannah Brandon
67dcadafb2cd7d4f3a26603b
Prithvijit Chattopadhyay
admin_assigned
2025-03-21T12:55:45.738Z
628d451386d23ad1560882c4
https://cdn-avatars.huggingface.co/v1/production/uploads/628d451386d23ad1560882c4/UMxez0DEvX5qdP5ddqi-8.png
Prithvijit Chattopadhyay
user
prithv1
67dcadafb2cd7d4f3a26603c
Huayu Chen
admin_assigned
2025-03-21T12:56:00.301Z
6630f87ee53fcb71c3887df0
/avatars/50191a3d45bebf90cf08df09477e95db.svg
HuayuChen
user
HuayuChen
67dcadafb2cd7d4f3a26603d
Jinju Chu
67dcadafb2cd7d4f3a26603e
Yin Cui
claimed_verified
2025-03-24T08:54:25.189Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
67dcadafb2cd7d4f3a26603f
Jenna Diamond
67dcadafb2cd7d4f3a266040
Yifan Ding
admin_assigned
2025-03-21T12:56:21.932Z
5e08d1e0fcf41d740b69964f
/avatars/926fef14351d247a0287f6f9dc9e3cb6.svg
Yifan Ding
user
YifanDing
67dcadafb2cd7d4f3a266041
Francesco Ferroni
claimed_verified
2025-03-21T11:41:11.413Z
6353d3e95eac2d2efa7501f9
/avatars/4b063f54000bed4bfb1bfcc3cde1a09e.svg
Francesco Ferroni
user
fferroni
67dcadafb2cd7d4f3a266042
Rama Govindaraju
67dcadafb2cd7d4f3a266043
Jinwei Gu
admin_assigned
2025-03-21T12:56:32.134Z
647e8118770c299e56fc2bc8
/avatars/adf80f3473dda42450148789ae5c208f.svg
Jinwei Gu
user
jwgu
67dcadafb2cd7d4f3a266044
Siddharth Gururani
admin_assigned
2025-03-21T12:56:41.456Z
679c387ac4fd876ea127b49f
/avatars/605f04dce6c2bf0f428642650d6642f5.svg
Siddharth Gururani
user
siddfisher
67dcadafb2cd7d4f3a266045
Imad El Hanafi
67dcadafb2cd7d4f3a266046
Zekun Hao
admin_assigned
2025-03-21T12:56:51.384Z
6721a3afef642d85f1f6eed5
/avatars/1689e9ef452a7b71189b5d81fadeff01.svg
Zekun Hao
user
zekunhao
67dcadafb2cd7d4f3a266047
Jacob Huffman
67dcadafb2cd7d4f3a266048
Jingyi Jin
admin_assigned
2025-03-21T12:57:01.419Z
678ec9e693649b2688c1d2ad
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tLzqFzkmPu8o-A8PtViGO.png
Jingyi Jin
user
jingyijin03
67dcadafb2cd7d4f3a266049
Brendan Johnson
admin_assigned
2025-03-21T12:57:09.652Z
646f8b3d850a938d6c537775
https://cdn-avatars.huggingface.co/v1/production/uploads/646f8b3d850a938d6c537775/-bO4-nu6KocS9Wa1gcX4H.png
Brendan Johnson
user
BrendanBJohnson
67dcadafb2cd7d4f3a26604a
Rizwan Khan
67dcadafb2cd7d4f3a26604b
George Kurian
admin_assigned
2025-03-21T12:57:31.615Z
66d05ee16447c6c45f54d65d
/avatars/21df19be6a2e3426c858ad373e44f39b.svg
George Kurian
user
Duke09
67dcadafb2cd7d4f3a26604c
Elena Lantz
67dcadafb2cd7d4f3a26604d
Nayeon Lee
67dcadafb2cd7d4f3a26604e
Zhaoshuo Li
admin_assigned
2025-03-21T12:57:42.436Z
647f13d85e1bc4753743f79e
/avatars/13a2a7fbd55d2e089fcf9ff0f9a41635.svg
Zhaoshuo Li
user
mli0603
67dcadafb2cd7d4f3a26604f
Xuan Li
67dcadafb2cd7d4f3a266050
Tsung-Yi Lin
67dcadafb2cd7d4f3a266051
Yen-Chen Lin
admin_assigned
2025-03-21T12:58:29.625Z
654c025dfda290a640090714
/avatars/83b730d2ed91f824e3f35e1999f5377f.svg
Yen-Chen Lin
user
yenchenl-nvidia
67dcadafb2cd7d4f3a266052
Ming-Yu Liu
admin_assigned
2025-03-21T12:58:21.711Z
62f049afdf4b93aad5c7f2d6
/avatars/e272e58ad996733d7098e50248e5b57e.svg
Ming-Yu Liu
user
mingyuliutw
67dcadafb2cd7d4f3a266053
Andrew Mathau
67dcadafb2cd7d4f3a266054
Yun Ni
67dcadafb2cd7d4f3a266055
Lindsey Pavao
67dcadafb2cd7d4f3a266056
Wei Ping
67dcadafb2cd7d4f3a266057
David W. Romero
claimed_verified
2025-03-24T08:54:37.389Z
6734efd27d73f338440d2420
/avatars/390187e6670dea73987840ba2de8706d.svg
David W Romero
user
dwromero-nv
67dcadafb2cd7d4f3a266058
Misha Smelyanskiy
admin_assigned
2025-03-21T12:58:13.481Z
6606f3def4ab651901fe8602
/avatars/bacd64657a0154d2a319c9b411ef9fe4.svg
Misha Smelyanskiy
user
Msmelyan
67dcadafb2cd7d4f3a266059
Shuran Song
67dcadafb2cd7d4f3a26605a
Lyne Tchapmi
67dcadafb2cd7d4f3a26605b
Andrew Z. Wang
claimed_verified
2025-06-02T07:52:20.071Z
6690456b54b460a2a6ad0817
/avatars/c0ca52463ed2157be6390daed2a484b5.svg
Andrew Z. Wang
user
andrewzwang
67dcadafb2cd7d4f3a26605c
Boxin Wang
67dcadafb2cd7d4f3a26605d
Haoxiang Wang
67dcadafb2cd7d4f3a26605e
Fangyin Wei
67dcadafb2cd7d4f3a26605f
Jiashu Xu
67dcadafb2cd7d4f3a266060
Yao Xu
67dcadafb2cd7d4f3a266061
Xiaodong Yang
admin_assigned
2025-03-21T12:58:02.852Z
64ab727051756fb15c275c5d
/avatars/26f77a1f81f0f4703eb87e542cd81fa9.svg
Yangxiaodong
user
Xiaodongyang
67dcadafb2cd7d4f3a266062
Zhuolin Yang
claimed_verified
2025-03-21T11:41:13.584Z
67d75b0117c2acac528f47b6
/avatars/619aacd1a619aab64de3499ac3ee2229.svg
Zhuolin Yang
user
zhuoliny
67dcadafb2cd7d4f3a266063
Xiaohui Zeng
admin_assigned
2025-03-21T12:57:54.702Z
64d9cf178767727dff1da0c7
/avatars/c2ec61b511754a8d312e2db3229215ec.svg
Xiaohui  ZENG
user
eccony
67dcadafb2cd7d4f3a266064
Zhe Zhang
Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
Physical AI systems need to perceive, understand, and perform complex actions
in the physical world. In this paper, we present the Cosmos-Reason1 models that
can understand the physical world and generate appropriate embodied decisions
(e.g., next step action) in natural language through long chain-of-thought
reasoning processes. We begin by defining key capabilities for Physical AI
reasoning, with a focus on physical common sense and embodied reasoning. To
represent physical common sense, we use a hierarchical ontology that captures
fundamental knowledge about space, time, and physics. For embodied reasoning,
we rely on a two-dimensional ontology that generalizes across different
physical embodiments. Building on these capabilities, we develop two multimodal
large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B. We curate data
and train our models in four stages: vision pre-training, general supervised
fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
as the post-training. To evaluate our models, we build comprehensive benchmarks
for physical common sense and embodied reasoning according to our ontologies.
Evaluation results show that Physical AI SFT and reinforcement learning bring
significant improvements. To facilitate the development of Physical AI, we will
make our code and pre-trained models available under the NVIDIA Open Model
License at https://github.com/nvidia-cosmos/cosmos-reason1.
Cosmos-Reason1 models, using hierarchical and two-dimensional ontologies for physical common sense and embodied reasoning, generate embodied decisions through multimodal large language models trained in vision and Physical AI stages.
Physical AI reasoning chain-of-thought hierarchical ontology two-dimensional ontology physical common sense embodied reasoning multimodal large language models vision pre-training supervised fine-tuning Physical AI SFT Physical AI reinforcement learning benchmarks