Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography
Metalenses offer significant potential for ultra-compact computational
imaging but face challenges from complex optical degradation and computational
restoration difficulties. Existing methods typically rely on precise optical
calibration or massive paired datasets, which are non-trivial for real-world
imaging systems. Furthermore, a lack of control over the inference process
often results in undesirable hallucinated artifacts. We introduce
Degradation-Modeled Multipath Diffusion for tunable metalens photography,
leveraging powerful natural image priors from pretrained models instead of
large datasets. Our framework uses positive, neutral, and negative-prompt paths
to balance high-frequency detail generation, structural fidelity, and
suppression of metalens-specific degradation, alongside pseudo data
augmentation. A tunable decoder enables controlled trade-offs between fidelity
and perceptual quality. Additionally, a spatially varying degradation-aware
attention (SVDA) module adaptively models complex optical and sensor-induced
degradation. Finally, we design and build a millimeter-scale MetaCamera for
real-world validation. Extensive results show that our approach outperforms
state-of-the-art methods, achieving high-fidelity and sharp image
reconstruction. More materials: https://dmdiff.github.io/.
6863b259588cea0da970c9bf
Jianing Zhang
6863b259588cea0da970c9c0
Jiayi Zhu
6863b259588cea0da970c9c1
Feiyu Ji
6863b259588cea0da970c9c2
Xiaokang Yang
6863b259588cea0da970c9c3
Xiaoyun Yuan
claimed_verified
2025-07-01T10:12:11.708Z
67761e674467879a54b4624a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hbniDcGynaZIUZTNSgc2G.jpeg
Xiaoyun Yuan
user
XiaoyunYuan
Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography
Metalenses offer significant potential for ultra-compact computational
imaging but face challenges from complex optical degradation and computational
restoration difficulties. Existing methods typically rely on precise optical
calibration or massive paired datasets, which are non-trivial for real-world
imaging systems. Furthermore, a lack of control over the inference process
often results in undesirable hallucinated artifacts. We introduce
Degradation-Modeled Multipath Diffusion for tunable metalens photography,
leveraging powerful natural image priors from pretrained models instead of
large datasets. Our framework uses positive, neutral, and negative-prompt paths
to balance high-frequency detail generation, structural fidelity, and
suppression of metalens-specific degradation, alongside pseudo data
augmentation. A tunable decoder enables controlled trade-offs between fidelity
and perceptual quality. Additionally, a spatially varying degradation-aware
attention (SVDA) module adaptively models complex optical and sensor-induced
degradation. Finally, we design and build a millimeter-scale MetaCamera for
real-world validation. Extensive results show that our approach outperforms
state-of-the-art methods, achieving high-fidelity and sharp image
reconstruction. More materials: https://dmdiff.github.io/.
The proposed Degradation-Modeled Multipath Diffusion framework improves metalens image quality by using natural image priors and specific modules to balance detail, fidelity, and perceptual quality while addressing optical degradation.
multipath diffusion natural image priors positive-prompt paths neutral-prompt paths negative-prompt paths pseudo data augmentation tunable decoder spatially varying degradation-aware attention SVDA module high-fidelity image reconstruction