NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation
  Learning for Neural Radiance Fields
Neural fields excel in computer vision and robotics due to their ability to
understand the 3D visual world such as inferring semantics, geometry, and
dynamics. Given the capabilities of neural fields in densely representing a 3D
scene from 2D images, we ask the question: Can we scale their self-supervised
pretraining, specifically using masked autoencoders, to generate effective 3D
representations from posed RGB images. Owing to the astounding success of
extending transformers to novel data modalities, we employ standard 3D Vision
Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
volumetric grid as a dense input to the transformer, contrasting it with other
3D representations such as pointclouds where the information density can be
uneven, and the representation is irregular. Due to the difficulty of applying
masked autoencoders to an implicit representation, such as NeRF, we opt for
extracting an explicit representation that canonicalizes scenes across domains
by employing the camera trajectory for sampling. Our goal is made possible by
masking random patches from NeRF's radiance and density grid and employing a
standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
the model can learn the semantic and spatial structure of complete scenes. We
pretrain this representation at scale on our proposed curated posed-RGB data,
totaling over 1.8 million images. Once pretrained, the encoder is used for
effective 3D transfer learning. Our novel self-supervised pretraining for
NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
scene understanding baselines on Front3D and ScanNet datasets with an absolute
performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.
66abcff29a9903e19a6c71f4
Muhammad Zubair Irshad
claimed_verified
2024-08-03T08:25:56.137Z
62e458d33051028b542be2a0
https://cdn-avatars.huggingface.co/v1/production/uploads/1659131510466-noauth.jpeg
Zubair Irshad
user
mirshad7
66abcff29a9903e19a6c71f5
Sergey Zakharov
admin_assigned
2024-08-02T16:02:27.368Z
630fdeaba119d49bc1e77c13
/avatars/5f5f3403278da36a3deb9df67eeb8c10.svg
Sergey Zakharov
user
zakharos
66abcff29a9903e19a6c71f6
Vitor Guizilini
66abcff29a9903e19a6c71f7
Adrien Gaidon
admin_assigned
2024-08-02T16:02:01.674Z
64af547e46a29208dafca2f3
/avatars/daff86330d644319c10347d230ab2452.svg
Adrien Gaidon
user
adriengaidon
66abcff29a9903e19a6c71f8
Zsolt Kira
admin_assigned
2024-08-02T16:01:55.140Z
64eeaa32f3ef4a52e2b3f5f6
/avatars/00ce423bee728a5123cf4bad422b1384.svg
Zsolt Kira
user
zsoltkira
66abcff29a9903e19a6c71f9
Rares Ambrus
admin_assigned
2024-08-02T16:01:28.544Z
633b349b9f5846dc43045bf9
/avatars/8b0dc52da2d01e22b981fc24257626e9.svg
Rares Ambrus
user
raresambrus-lbm
NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation
  Learning for Neural Radiance Fields
Neural fields excel in computer vision and robotics due to their ability to
understand the 3D visual world such as inferring semantics, geometry, and
dynamics. Given the capabilities of neural fields in densely representing a 3D
scene from 2D images, we ask the question: Can we scale their self-supervised
pretraining, specifically using masked autoencoders, to generate effective 3D
representations from posed RGB images. Owing to the astounding success of
extending transformers to novel data modalities, we employ standard 3D Vision
Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
volumetric grid as a dense input to the transformer, contrasting it with other
3D representations such as pointclouds where the information density can be
uneven, and the representation is irregular. Due to the difficulty of applying
masked autoencoders to an implicit representation, such as NeRF, we opt for
extracting an explicit representation that canonicalizes scenes across domains
by employing the camera trajectory for sampling. Our goal is made possible by
masking random patches from NeRF's radiance and density grid and employing a
standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
the model can learn the semantic and spatial structure of complete scenes. We
pretrain this representation at scale on our proposed curated posed-RGB data,
totaling over 1.8 million images. Once pretrained, the encoder is used for
effective 3D transfer learning. Our novel self-supervised pretraining for
NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
scene understanding baselines on Front3D and ScanNet datasets with an absolute
performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.
NeRF-MAE, a self-supervised pretraining approach using masked autoencoders and 3D Swin Transformers, enhances NeRF for effective 3D representation and improves scene understanding across datasets.
neural fields NeRF masked autoencoders 3D Vision Transformers volumetric grid pointclouds implicit representation explicit representation radiance grid density grid 3D Swin Transformer NeRF-MAE 3D transfer learning Front3D ScanNet 3D object detection