Neural Gaffer: Relighting Any Object via Diffusion
Single-image relighting is a challenging task that involves reasoning about
the complex interplay between geometry, materials, and lighting. Many prior
methods either support only specific categories of images, such as portraits,
or require special capture conditions, like using a flashlight. Alternatively,
some methods explicitly decompose a scene into intrinsic components, such as
normals and BRDFs, which can be inaccurate or under-expressive. In this work,
we propose a novel end-to-end 2D relighting diffusion model, called Neural
Gaffer, that takes a single image of any object and can synthesize an accurate,
high-quality relit image under any novel environmental lighting condition,
simply by conditioning an image generator on a target environment map, without
an explicit scene decomposition. Our method builds on a pre-trained diffusion
model, and fine-tunes it on a synthetic relighting dataset, revealing and
harnessing the inherent understanding of lighting present in the diffusion
model. We evaluate our model on both synthetic and in-the-wild Internet imagery
and demonstrate its advantages in terms of generalization and accuracy.
Moreover, by combining with other generative methods, our model enables many
downstream 2D tasks, such as text-based relighting and object insertion. Our
model can also operate as a strong relighting prior for 3D tasks, such as
relighting a radiance field.
6669d80f1537c1bb0f8eb068
Haian Jin
admin_assigned
2024-06-13T09:35:04.055Z
64a04ad150251e408af12e01
/avatars/e4f630616a16a7fc691b5275bb62f2aa.svg
Haian Jin
user
coast01
6669d80f1537c1bb0f8eb069
Yuan Li
6669d80f1537c1bb0f8eb06a
Fujun Luan
admin_assigned
2024-06-13T09:35:26.325Z
662ae0aa4f711ee4e1e17ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/662ae0aa4f711ee4e1e17ffc/nEi6tv8f6DIhTauwHpw_C.jpeg
Fujun Luan
user
luanfujun
6669d80f1537c1bb0f8eb06b
Yuanbo Xiangli
admin_assigned
2024-06-13T09:35:32.164Z
651edd0faa888068acb5d680
/avatars/d5e03e8bc1d22bc67a3e684b52dc5598.svg
Yuanbo Xiangli
user
amberkk
6669d80f1537c1bb0f8eb06c
Sai Bi
admin_assigned
2024-06-13T09:35:38.403Z
66303560597dc7c535168f07
/avatars/a69d52a4a198c6c8451c1d137031bf07.svg
Sai Bi
user
saibi
6669d80f1537c1bb0f8eb06d
Kai Zhang
6669d80f1537c1bb0f8eb06e
Zexiang Xu
admin_assigned
2024-06-13T09:35:56.113Z
65565b1b8616d8d740abb976
/avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
Zexiang Xu
user
zexiangxu
6669d80f1537c1bb0f8eb06f
Jin Sun
6669d80f1537c1bb0f8eb070
Noah Snavely
admin_assigned
2024-06-13T09:36:17.215Z
632b79ca0a9dc02985af9e0b
/avatars/e2ca585398a3212b2956a14804d0ba67.svg
Noah Snavely
user
jimantha
Neural Gaffer: Relighting Any Object via Diffusion
Single-image relighting is a challenging task that involves reasoning about
the complex interplay between geometry, materials, and lighting. Many prior
methods either support only specific categories of images, such as portraits,
or require special capture conditions, like using a flashlight. Alternatively,
some methods explicitly decompose a scene into intrinsic components, such as
normals and BRDFs, which can be inaccurate or under-expressive. In this work,
we propose a novel end-to-end 2D relighting diffusion model, called Neural
Gaffer, that takes a single image of any object and can synthesize an accurate,
high-quality relit image under any novel environmental lighting condition,
simply by conditioning an image generator on a target environment map, without
an explicit scene decomposition. Our method builds on a pre-trained diffusion
model, and fine-tunes it on a synthetic relighting dataset, revealing and
harnessing the inherent understanding of lighting present in the diffusion
model. We evaluate our model on both synthetic and in-the-wild Internet imagery
and demonstrate its advantages in terms of generalization and accuracy.
Moreover, by combining with other generative methods, our model enables many
downstream 2D tasks, such as text-based relighting and object insertion. Our
model can also operate as a strong relighting prior for 3D tasks, such as
relighting a radiance field.
A novel end-to-end 2D relighting diffusion model, Neural Gaffer, synthesizes high-quality relit images under various lighting conditions without scene decomposition by fine-tuning a pre-trained diffusion model.
end-to-end relighting diffusion model Neural Gaffer pre-trained diffusion model synthetic relighting dataset environment map text-based relighting object insertion relighting prior radiance field