3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World
  Model
Manipulation has long been a challenging task for robots, while humans can
effortlessly perform complex interactions with objects, such as hanging a cup
on the mug rack. A key reason is the lack of a large and uniform dataset for
teaching robots manipulation skills. Current robot datasets often record robot
action in different action spaces within a simple scene. This hinders the robot
to learn a unified and robust action representation for different robots within
diverse scenes. Observing how humans understand a manipulation task, we find
that understanding how the objects should move in the 3D space is a critical
clue for guiding actions. This clue is embodiment-agnostic and suitable for
both humans and different robots. Motivated by this, we aim to learn a 3D flow
world model from both human and robot manipulation data. This model predicts
the future movement of the interacting objects in 3D space, guiding action
planning for manipulation. Specifically, we synthesize a large-scale 3D optical
flow dataset, named ManiFlow-110k, through a moving object auto-detect
pipeline. A video diffusion-based world model then learns manipulation physics
from these data, generating 3D optical flow trajectories conditioned on
language instructions. With the generated 3D object optical flow, we propose a
flow-guided rendering mechanism, which renders the predicted final state and
leverages GPT-4o to assess whether the predicted flow aligns with the task
description. This equips the robot with a closed-loop planning ability.
Finally, we consider the predicted 3D optical flow as constraints for an
optimization policy to determine a chunk of robot actions for manipulation.
Extensive experiments demonstrate strong generalization across diverse robotic
manipulation tasks and reliable cross-embodiment adaptation without
hardware-specific training.
684637733ec10bdd8ab4da66
Hongyan Zhi
claimed_verified
2025-06-09T10:11:27.821Z
674b2406591d7232820252cd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bhDlUVkFQ66yt3BEVs6WU.png
Hongyan Zhi
user
Hoyard
684637733ec10bdd8ab4da67
Peihao Chen
684637733ec10bdd8ab4da68
Siyuan Zhou
684637733ec10bdd8ab4da69
Yubo Dong
684637733ec10bdd8ab4da6a
Quanxi Wu
684637733ec10bdd8ab4da6b
Lei Han
684637733ec10bdd8ab4da6c
Mingkui Tan
3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World
  Model
Manipulation has long been a challenging task for robots, while humans can
effortlessly perform complex interactions with objects, such as hanging a cup
on the mug rack. A key reason is the lack of a large and uniform dataset for
teaching robots manipulation skills. Current robot datasets often record robot
action in different action spaces within a simple scene. This hinders the robot
to learn a unified and robust action representation for different robots within
diverse scenes. Observing how humans understand a manipulation task, we find
that understanding how the objects should move in the 3D space is a critical
clue for guiding actions. This clue is embodiment-agnostic and suitable for
both humans and different robots. Motivated by this, we aim to learn a 3D flow
world model from both human and robot manipulation data. This model predicts
the future movement of the interacting objects in 3D space, guiding action
planning for manipulation. Specifically, we synthesize a large-scale 3D optical
flow dataset, named ManiFlow-110k, through a moving object auto-detect
pipeline. A video diffusion-based world model then learns manipulation physics
from these data, generating 3D optical flow trajectories conditioned on
language instructions. With the generated 3D object optical flow, we propose a
flow-guided rendering mechanism, which renders the predicted final state and
leverages GPT-4o to assess whether the predicted flow aligns with the task
description. This equips the robot with a closed-loop planning ability.
Finally, we consider the predicted 3D optical flow as constraints for an
optimization policy to determine a chunk of robot actions for manipulation.
Extensive experiments demonstrate strong generalization across diverse robotic
manipulation tasks and reliable cross-embodiment adaptation without
hardware-specific training.
A 3D flow world model learned from human and robot manipulation data, using video diffusion and GPT-4o, enables robots to perform diverse manipulation tasks with strong generalization and cross-embodiment adaptation.
3D flow world model moving object auto-detect pipeline video diffusion-based world model 3D optical flow dataset ManiFlow-110k 3D optical flow trajectories flow-guided rendering mechanism GPT-4o closed-loop planning optimization policy cross-embodiment adaptation