HyperFields: Towards Zero-Shot Generation of NeRFs from Text
We introduce HyperFields, a method for generating text-conditioned Neural
Radiance Fields (NeRFs) with a single forward pass and (optionally) some
fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns
a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF
distillation training, which distills scenes encoded in individual NeRFs into
one dynamic hypernetwork. These techniques enable a single network to fit over
a hundred unique scenes. We further demonstrate that HyperFields learns a more
general map between text and NeRFs, and consequently is capable of predicting
novel in-distribution and out-of-distribution scenes -- either zero-shot or
with a few finetuning steps. Finetuning HyperFields benefits from accelerated
convergence thanks to the learned general map, and is capable of synthesizing
novel scenes 5 to 10 times faster than existing neural optimization-based
methods. Our ablation experiments show that both the dynamic architecture and
NeRF distillation are critical to the expressivity of HyperFields.
653b29afd994e992e27b800d
Sudarshan Babu
claimed_verified
2023-10-27T09:12:43.718Z
6341baf206a304bdc432d28b
/avatars/31fe822d1cd05d9bd7009b97873cb527.svg
sudarshan babu
user
sudarshan94
653b29afd994e992e27b800e
Richard Liu
653b29afd994e992e27b800f
Avery Zhou
admin_assigned
2023-10-27T09:15:38.847Z
63433036aa45ed8ecf97da55
/avatars/63566616130bddfae399ff52d600e9e6.svg
Avery Zhou
user
averyzhou
653b29afd994e992e27b8010
Michael Maire
admin_assigned
2023-10-27T09:15:17.865Z
63dcf657f37111482522cd56
https://cdn-avatars.huggingface.co/v1/production/uploads/1675425351343-noauth.png
Michael Maire
user
MikeStudios
653b29afd994e992e27b8011
Greg Shakhnarovich
653b29afd994e992e27b8012
Rana Hanocka
admin_assigned
2023-10-27T09:15:28.844Z
630495ffce6b12280b193f30
/avatars/55b201131fa798c2a33d17dcab72fd1f.svg
Rana Hanocka
user
ranahanocka
HyperFields: Towards Zero-Shot Generation of NeRFs from Text
We introduce HyperFields, a method for generating text-conditioned Neural
Radiance Fields (NeRFs) with a single forward pass and (optionally) some
fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns
a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF
distillation training, which distills scenes encoded in individual NeRFs into
one dynamic hypernetwork. These techniques enable a single network to fit over
a hundred unique scenes. We further demonstrate that HyperFields learns a more
general map between text and NeRFs, and consequently is capable of predicting
novel in-distribution and out-of-distribution scenes -- either zero-shot or
with a few finetuning steps. Finetuning HyperFields benefits from accelerated
convergence thanks to the learned general map, and is capable of synthesizing
novel scenes 5 to 10 times faster than existing neural optimization-based
methods. Our ablation experiments show that both the dynamic architecture and
NeRF distillation are critical to the expressivity of HyperFields.
HyperFields Neural Radiance Fields (NeRFs) text token embeddings dynamic hypernetwork NeRF distillation training zero-shot in-distribution out-of-distribution accelerated convergence neural optimization-based methods ablation experiments dynamic architecture