PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape
  Prediction
We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing
a 3D object from a few unposed images even with little visual overlap, while
simultaneously estimating the relative camera poses in ~1.3 seconds on a single
A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention
blocks to exchange information between 3D object tokens and 2D image tokens; we
predict a coarse point cloud for each view, and then use a differentiable
Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge
amount of multi-view posed data of ~1M objects, PF-LRM shows strong
cross-dataset generalization ability, and outperforms baseline methods by a
large margin in terms of pose prediction accuracy and 3D reconstruction quality
on various unseen evaluation datasets. We also demonstrate our model's
applicability in downstream text/image-to-3D task with fast feed-forward
inference. Our project website is at: https://totoro97.github.io/pf-lrm .
655d9a35ba94da7590e78cef
Peng Wang
655d9a35ba94da7590e78cf0
Hao Tan
655d9a35ba94da7590e78cf1
Sai Bi
655d9a35ba94da7590e78cf2
Yinghao Xu
admin_assigned
2023-11-22T13:12:15.296Z
6555da9adcf410fd0753569c
/avatars/ac58a796bb54f334fdb475a4b75c4d27.svg
Yinghao Xu
user
justimyhxu
655d9a35ba94da7590e78cf3
Fujun Luan
655d9a35ba94da7590e78cf4
Kalyan Sunkavalli
655d9a35ba94da7590e78cf5
Wenping Wang
655d9a35ba94da7590e78cf6
Zexiang Xu
admin_assigned
2023-11-22T13:12:55.585Z
65565b1b8616d8d740abb976
/avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
Zexiang Xu
user
zexiangxu
655d9a35ba94da7590e78cf7
Kai Zhang
claimed_verified
2023-11-23T10:49:11.922Z
6351a882a4845f26d16a28d7
/avatars/4f8629fbf22faeb6511f4068f2828809.svg
Kai Zhang
user
Kai-46
PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape
  Prediction
We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing
a 3D object from a few unposed images even with little visual overlap, while
simultaneously estimating the relative camera poses in ~1.3 seconds on a single
A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention
blocks to exchange information between 3D object tokens and 2D image tokens; we
predict a coarse point cloud for each view, and then use a differentiable
Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge
amount of multi-view posed data of ~1M objects, PF-LRM shows strong
cross-dataset generalization ability, and outperforms baseline methods by a
large margin in terms of pose prediction accuracy and 3D reconstruction quality
on various unseen evaluation datasets. We also demonstrate our model's
applicability in downstream text/image-to-3D task with fast feed-forward
inference. Our project website is at: https://totoro97.github.io/pf-lrm .
A Pose-Free Large Reconstruction Model using self-attention for 3D reconstruction from few unposed images, estimating camera poses, and generalizing across datasets.
Pose-Free Large Reconstruction Model self-attention 3D object tokens 2D image tokens Perspective-n-Point PnP solver cross-dataset generalization text-to-3D image-to-3D feed-forward inference