Comprehensive Relighting: Generalizable and Consistent Monocular Human
  Relighting and Harmonization
This paper introduces Comprehensive Relighting, the first all-in-one approach
that can both control and harmonize the lighting from an image or video of
humans with arbitrary body parts from any scene. Building such a generalizable
model is extremely challenging due to the lack of dataset, restricting existing
image-based relighting models to a specific scenario (e.g., face or static
human). To address this challenge, we repurpose a pre-trained diffusion model
as a general image prior and jointly model the human relighting and background
harmonization in the coarse-to-fine framework. To further enhance the temporal
coherence of the relighting, we introduce an unsupervised temporal lighting
model that learns the lighting cycle consistency from many real-world videos
without any ground truth. In inference time, our temporal lighting module is
combined with the diffusion models through the spatio-temporal feature blending
algorithms without extra training; and we apply a new guided refinement as a
post-processing to preserve the high-frequency details from the input image. In
the experiments, Comprehensive Relighting shows a strong generalizability and
lighting temporal coherence, outperforming existing image-based human
relighting and harmonization methods.
67f35d8bdb1a843e1ceff38f
Junying Wang
67f35d8bdb1a843e1ceff390
Jingyuan Liu
67f35d8bdb1a843e1ceff391
Xin Sun
67f35d8bdb1a843e1ceff392
Krishna Kumar Singh
67f35d8bdb1a843e1ceff393
Zhixin Shu
67f35d8bdb1a843e1ceff394
He Zhang
67f35d8bdb1a843e1ceff395
Jimei Yang
67f35d8bdb1a843e1ceff396
Nanxuan Zhao
67f35d8bdb1a843e1ceff397
Tuanfeng Y. Wang
67f35d8bdb1a843e1ceff398
Simon S. Chen
67f35d8bdb1a843e1ceff399
Ulrich Neumann
67f35d8bdb1a843e1ceff39a
Jae Shin Yoon
Comprehensive Relighting: Generalizable and Consistent Monocular Human
  Relighting and Harmonization
This paper introduces Comprehensive Relighting, the first all-in-one approach
that can both control and harmonize the lighting from an image or video of
humans with arbitrary body parts from any scene. Building such a generalizable
model is extremely challenging due to the lack of dataset, restricting existing
image-based relighting models to a specific scenario (e.g., face or static
human). To address this challenge, we repurpose a pre-trained diffusion model
as a general image prior and jointly model the human relighting and background
harmonization in the coarse-to-fine framework. To further enhance the temporal
coherence of the relighting, we introduce an unsupervised temporal lighting
model that learns the lighting cycle consistency from many real-world videos
without any ground truth. In inference time, our temporal lighting module is
combined with the diffusion models through the spatio-temporal feature blending
algorithms without extra training; and we apply a new guided refinement as a
post-processing to preserve the high-frequency details from the input image. In
the experiments, Comprehensive Relighting shows a strong generalizability and
lighting temporal coherence, outperforming existing image-based human
relighting and harmonization methods.
Comprehensive Relighting uses diffusion models and temporal lighting modules to control and harmonize lighting in human images and videos with arbitrary body parts.
diffusion model image prior coarse-to-fine temporal lighting model lighting cycle consistency spatio-temporal feature blending guided refinement generalizability lighting temporal coherence