FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
We present FoundationPose, a unified foundation model for 6D object pose
estimation and tracking, supporting both model-based and model-free setups. Our
approach can be instantly applied at test-time to a novel object without
fine-tuning, as long as its CAD model is given, or a small number of reference
images are captured. We bridge the gap between these two setups with a neural
implicit representation that allows for effective novel view synthesis, keeping
the downstream pose estimation modules invariant under the same unified
framework. Strong generalizability is achieved via large-scale synthetic
training, aided by a large language model (LLM), a novel transformer-based
architecture, and contrastive learning formulation. Extensive evaluation on
multiple public datasets involving challenging scenarios and objects indicate
our unified approach outperforms existing methods specialized for each task by
a large margin. In addition, it even achieves comparable results to
instance-level methods despite the reduced assumptions. Project page:
https://nvlabs.github.io/FoundationPose/
657a6a4ec6240db29fee60cb
Bowen Wen
admin_assigned
2023-12-14T13:51:24.015Z
64aeee717b3b3386662e9e62
/avatars/8fd244261d6128a109c25ef8578e186b.svg
Bowen Wen
user
wenbowen
657a6a4ec6240db29fee60cc
Wei Yang
657a6a4ec6240db29fee60cd
Jan Kautz
657a6a4ec6240db29fee60ce
Stan Birchfield
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
We present FoundationPose, a unified foundation model for 6D object pose
estimation and tracking, supporting both model-based and model-free setups. Our
approach can be instantly applied at test-time to a novel object without
fine-tuning, as long as its CAD model is given, or a small number of reference
images are captured. We bridge the gap between these two setups with a neural
implicit representation that allows for effective novel view synthesis, keeping
the downstream pose estimation modules invariant under the same unified
framework. Strong generalizability is achieved via large-scale synthetic
training, aided by a large language model (LLM), a novel transformer-based
architecture, and contrastive learning formulation. Extensive evaluation on
multiple public datasets involving challenging scenarios and objects indicate
our unified approach outperforms existing methods specialized for each task by
a large margin. In addition, it even achieves comparable results to
instance-level methods despite the reduced assumptions. Project page:
https://nvlabs.github.io/FoundationPose/
FoundationPose, a unified foundation model using neural implicit representation and transformer architecture, outperforms specialized models in 6D object pose estimation and tracking.
neural implicit representation transformer-based architecture contrastive learning formulation