LLaNA: Large Language and NeRF Assistant
Multimodal Large Language Models (MLLMs) have demonstrated an excellent
understanding of images and 3D data. However, both modalities have shortcomings
in holistically capturing the appearance and geometry of objects. Meanwhile,
Neural Radiance Fields (NeRFs), which encode information within the weights of
a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly
widespread modality that simultaneously encodes the geometry and photorealistic
appearance of objects. This paper investigates the feasibility and
effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first
general-purpose NeRF-language assistant capable of performing new tasks such as
NeRF captioning and Q\&A. Notably, our method directly processes the weights of
the NeRF's MLP to extract information about the represented objects without the
need to render images or materialize 3D data structures. Moreover, we build a
dataset of NeRFs with text annotations for various NeRF-language tasks with no
human intervention. Based on this dataset, we develop a benchmark to evaluate
the NeRF understanding capability of our method. Results show that processing
NeRF weights performs favourably against extracting 2D or 3D representations
from NeRFs.
66713e14bb4aa4b6a9c83093
Andrea Amaduzzi
claimed_verified
2024-06-18T08:29:35.846Z
629f5f61ebbc7168c29f27c6
/avatars/1065d941089dd3c817702f62295c4431.svg
Andrea
user
andreamaduzzi
66713e14bb4aa4b6a9c83094
Pierluigi Zama Ramirez
admin_assigned
2024-06-18T14:57:34.514Z
651ea108fb142cd617e0aadf
/avatars/0c0cbf5b66388c088e7c654222db8bf9.svg
Pierluigi Zama Ramirez
user
pierlui92
66713e14bb4aa4b6a9c83095
Giuseppe Lisanti
extracted_confirmed
2024-09-03T10:36:35.797Z
6672a82e2a4d2e3e835f1f57
/avatars/520a11f13d30272557c85e20aafb1922.svg
Giuseppe Lisanti
user
giuseppelisanti
66713e14bb4aa4b6a9c83096
Samuele Salti
extracted_confirmed
2024-06-19T09:43:38.468Z
6672a802a3dc375df3fdfa3a
/avatars/4f37c3fd80080d1c677ebc4a1c565f11.svg
Samuele Salti
user
samuelesalti
66713e14bb4aa4b6a9c83097
Luigi Di Stefano
extracted_pending
2024-06-20T06:34:45.013Z
6673cd4056cf3c3548349f16
/avatars/71337fb4bad4e89d8fb8584535017cac.svg
Luigi Di Stefano
user
ldistef
LLaNA: Large Language and NeRF Assistant
Multimodal Large Language Models (MLLMs) have demonstrated an excellent
understanding of images and 3D data. However, both modalities have shortcomings
in holistically capturing the appearance and geometry of objects. Meanwhile,
Neural Radiance Fields (NeRFs), which encode information within the weights of
a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly
widespread modality that simultaneously encodes the geometry and photorealistic
appearance of objects. This paper investigates the feasibility and
effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first
general-purpose NeRF-language assistant capable of performing new tasks such as
NeRF captioning and Q\&A. Notably, our method directly processes the weights of
the NeRF's MLP to extract information about the represented objects without the
need to render images or materialize 3D data structures. Moreover, we build a
dataset of NeRFs with text annotations for various NeRF-language tasks with no
human intervention. Based on this dataset, we develop a benchmark to evaluate
the NeRF understanding capability of our method. Results show that processing
NeRF weights performs favourably against extracting 2D or 3D representations
from NeRFs.
LLaNA, an MLLM integrated with NeRFs, effectively performs tasks such as NeRF captioning and Q&A by processing NeRF weights directly, outperforming 2D or 3D representation extraction methods.
Multimodal Large Language Models MLLMs Neural Radiance Fields NeRFs Multi-Layer Perceptron MLP NeRF captioning NeRF Q&A benchmark NeRF understanding capability