LuxDiT: Lighting Estimation with Video Diffusion Transformer
Estimating scene lighting from a single image or video remains a longstanding
challenge in computer vision and graphics. Learning-based approaches are
constrained by the scarcity of ground-truth HDR environment maps, which are
expensive to capture and limited in diversity. While recent generative models
offer strong priors for image synthesis, lighting estimation remains difficult
due to its reliance on indirect visual cues, the need to infer global
(non-local) context, and the recovery of high-dynamic-range outputs. We propose
LuxDiT, a novel data-driven approach that fine-tunes a video diffusion
transformer to generate HDR environment maps conditioned on visual input.
Trained on a large synthetic dataset with diverse lighting conditions, our
model learns to infer illumination from indirect visual cues and generalizes
effectively to real-world scenes. To improve semantic alignment between the
input and the predicted environment map, we introduce a low-rank adaptation
finetuning strategy using a collected dataset of HDR panoramas. Our method
produces accurate lighting predictions with realistic angular high-frequency
details, outperforming existing state-of-the-art techniques in both
quantitative and qualitative evaluations.
68be8f39c123124955ef615a
Ruofan Liang
68be8f39c123124955ef615b
Kai He
68be8f39c123124955ef615c
Zan Gojcic
68be8f39c123124955ef615d
Igor Gilitschenski
68be8f39c123124955ef615e
Sanja Fidler
68be8f39c123124955ef615f
Nandita Vijaykumar
68be8f39c123124955ef6160
Zian Wang
LuxDiT: Lighting Estimation with Video Diffusion Transformer
Estimating scene lighting from a single image or video remains a longstanding
challenge in computer vision and graphics. Learning-based approaches are
constrained by the scarcity of ground-truth HDR environment maps, which are
expensive to capture and limited in diversity. While recent generative models
offer strong priors for image synthesis, lighting estimation remains difficult
due to its reliance on indirect visual cues, the need to infer global
(non-local) context, and the recovery of high-dynamic-range outputs. We propose
LuxDiT, a novel data-driven approach that fine-tunes a video diffusion
transformer to generate HDR environment maps conditioned on visual input.
Trained on a large synthetic dataset with diverse lighting conditions, our
model learns to infer illumination from indirect visual cues and generalizes
effectively to real-world scenes. To improve semantic alignment between the
input and the predicted environment map, we introduce a low-rank adaptation
finetuning strategy using a collected dataset of HDR panoramas. Our method
produces accurate lighting predictions with realistic angular high-frequency
details, outperforming existing state-of-the-art techniques in both
quantitative and qualitative evaluations.
LuxDiT, a video diffusion transformer fine-tuned with low-rank adaptation, generates accurate HDR environment maps from visual input, outperforming existing methods.
video diffusion transformer HDR environment maps low-rank adaptation synthetic dataset indirect visual cues global context high-dynamic-range outputs HDR panoramas angular high-frequency details