Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography
Metalenses offer significant potential for ultra-compact computational
imaging but face challenges from complex optical degradation and computational
restoration difficulties. Existing methods typically rely on precise optical
calibration or massive paired datasets, which are non-trivial for real-world
imaging systems. Furthermore, a lack of control over the inference process
often results in undesirable hallucinated artifacts. We introduce
Degradation-Modeled Multipath Diffusion for tunable metalens photography,
leveraging powerful natural image priors from pretrained models instead of
large datasets. Our framework uses positive, neutral, and negative-prompt paths
to balance high-frequency detail generation, structural fidelity, and
suppression of metalens-specific degradation, alongside pseudo data
augmentation. A tunable decoder enables controlled trade-offs between fidelity
and perceptual quality. Additionally, a spatially varying degradation-aware
attention (SVDA) module adaptively models complex optical and sensor-induced
degradation. Finally, we design and build a millimeter-scale MetaCamera for
real-world validation. Extensive results show that our approach outperforms
state-of-the-art methods, achieving high-fidelity and sharp image
reconstruction. More materials: https://dmdiff.github.io/.
6863b259588cea0da970c9bf
Jianing Zhang
6863b259588cea0da970c9c0
Jiayi Zhu
6863b259588cea0da970c9c1
Feiyu Ji
6863b259588cea0da970c9c2
Xiaokang Yang
6863b259588cea0da970c9c3
Xiaoyun Yuan
claimed_verified
2025-07-01T10:12:11.708Z
67761e674467879a54b4624a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hbniDcGynaZIUZTNSgc2G.jpeg
Xiaoyun Yuan
user
XiaoyunYuan
Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography
Metalenses offer significant potential for ultra-compact computational
imaging but face challenges from complex optical degradation and computational
restoration difficulties. Existing methods typically rely on precise optical
calibration or massive paired datasets, which are non-trivial for real-world
imaging systems. Furthermore, a lack of control over the inference process
often results in undesirable hallucinated artifacts. We introduce
Degradation-Modeled Multipath Diffusion for tunable metalens photography,
leveraging powerful natural image priors from pretrained models instead of
large datasets. Our framework uses positive, neutral, and negative-prompt paths
to balance high-frequency detail generation, structural fidelity, and
suppression of metalens-specific degradation, alongside pseudo data
augmentation. A tunable decoder enables controlled trade-offs between fidelity
and perceptual quality. Additionally, a spatially varying degradation-aware
attention (SVDA) module adaptively models complex optical and sensor-induced
degradation. Finally, we design and build a millimeter-scale MetaCamera for
real-world validation. Extensive results show that our approach outperforms
state-of-the-art methods, achieving high-fidelity and sharp image
reconstruction. More materials: https://dmdiff.github.io/.
A novel framework using degradation-modeled multipath diffusion and spatially varying degradation-aware attention improves image reconstruction for metalenses with high fidelity and sharpness.
multipath diffusion natural image priors positive-prompt paths neutral-prompt paths negative-prompt paths pseudo data augmentation tunable decoder spatially varying degradation-aware attention SVDA module MetaCamera