Generative Perception of Shape and Material from Differential Motion
Xinran Han, Ko Nishino, Todd Zickler
Perceiving the shape and material of an object from a single image is inherently ambiguous, especially when lighting is unknown and unconstrained. Despite this, humans can often disentangle shape and material, and when they are uncertain, they often move their head slightly or rotate the object to help resolve the ambiguities. Inspired by this behavior, we introduce a novel conditional denoising-diffusion model that generates samples of shape-and-material maps from a given short input video of an object undergoing differential motions. Our parameter-efficient architecture allows training directly in pixel-space and joint, generative disentanglement of multiple object attributes simultaneously. Trained on a modest number of synthetic object-motion videos with supervision on shape and material, the model exhibits compelling emergent properties: for static observations, it produces diverse, multimodal predictions of plausible shape-and-material maps that capture the inherent ambiguities; and when objects move, the distributions quickly converge to more accurate explanations. Meanwhile, it produces high-quality shape-and-material estimates on less ambiguous, real-world objects. By moving beyond single view to continuous observations, our work suggests an avenue of generative perception for improving visual reasoning for physically-embodied systems.