ControlMat: A Controlled Generative Approach to Material Capture
Material reconstruction from a photograph is a key component of 3D content
creation democratization. We propose to formulate this ill-posed problem as a
controlled synthesis one, leveraging the recent progress in generative deep
networks. We present ControlMat, a method which, given a single photograph with
uncontrolled illumination as input, conditions a diffusion model to generate
plausible, tileable, high-resolution physically-based digital materials. We
carefully analyze the behavior of diffusion models for multi-channel outputs,
adapt the sampling process to fuse multi-scale information and introduce rolled
diffusion to enable both tileability and patched diffusion for high-resolution
outputs. Our generative approach further permits exploration of a variety of
materials which could correspond to the input image, mitigating the unknown
lighting conditions. We show that our approach outperforms recent inference and
latent-space-optimization methods, and carefully validate our diffusion process
design choices. Supplemental materials and additional details are available at:
https://gvecchio.com/controlmat/.
64f82f091435cc7faa40db49
Giuseppe Vecchio
claimed_verified
2023-09-06T11:31:47.735Z
643fa1c718afbc4d1f3e5e94
https://cdn-avatars.huggingface.co/v1/production/uploads/643fa1c718afbc4d1f3e5e94/_cg3HHuycPGHnexJHbfwP.jpeg
Giuseppe Vecchio
user
gvecchio
64f82f091435cc7faa40db4a
Rosalie Martin
64f82f091435cc7faa40db4b
Arthur Roullier
64f82f091435cc7faa40db4c
Adrien Kaiser
extracted_confirmed
2023-09-06T09:58:00.761Z
63370068612e2c3c46d99434
/avatars/cf91b83f058611046ce8e5e06a12cb13.svg
Adrien Kaiser
user
akaiser
64f82f091435cc7faa40db4d
Romain Rouffet
64f82f091435cc7faa40db4e
Valentin Deschaintre
extracted_confirmed
2023-09-06T08:35:32.438Z
63e0dc30ab3fb81c39816ce7
/avatars/a30c8ca8850ae120dd409b6f7d204493.svg
Valentin Deschaintre
user
deschain
64f82f091435cc7faa40db4f
Tamy Boubekeur
ControlMat: A Controlled Generative Approach to Material Capture
Material reconstruction from a photograph is a key component of 3D content
creation democratization. We propose to formulate this ill-posed problem as a
controlled synthesis one, leveraging the recent progress in generative deep
networks. We present ControlMat, a method which, given a single photograph with
uncontrolled illumination as input, conditions a diffusion model to generate
plausible, tileable, high-resolution physically-based digital materials. We
carefully analyze the behavior of diffusion models for multi-channel outputs,
adapt the sampling process to fuse multi-scale information and introduce rolled
diffusion to enable both tileability and patched diffusion for high-resolution
outputs. Our generative approach further permits exploration of a variety of
materials which could correspond to the input image, mitigating the unknown
lighting conditions. We show that our approach outperforms recent inference and
latent-space-optimization methods, and carefully validate our diffusion process
design choices. Supplemental materials and additional details are available at:
https://gvecchio.com/controlmat/.
A method called ControlMat uses a diffusion model to generate high-resolution, tileable digital materials from a single photograph, addressing challenges of uncontrolled lighting.
diffusion model tileable high-resolution physically-based digital materials rolled diffusion patched diffusion multi-channel outputs multi-scale information generative approach latent-space-optimization methods