PixelHacker: Image Inpainting with Structural and Semantic Consistency
Image inpainting is a fundamental research area between image editing and
image generation. Recent state-of-the-art (SOTA) methods have explored novel
attention mechanisms, lightweight architectures, and context-aware modeling,
demonstrating impressive performance. However, they often struggle with complex
structure (e.g., texture, shape, spatial relations) and semantics (e.g., color
consistency, object restoration, and logical correctness), leading to artifacts
and inappropriate generation. To address this challenge, we design a simple yet
effective inpainting paradigm called latent categories guidance, and further
propose a diffusion-based model named PixelHacker. Specifically, we first
construct a large dataset containing 14 million image-mask pairs by annotating
foreground and background (potential 116 and 21 categories, respectively).
Then, we encode potential foreground and background representations separately
through two fixed-size embeddings, and intermittently inject these features
into the denoising process via linear attention. Finally, by pre-training on
our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker.
Extensive experiments show that PixelHacker comprehensively outperforms the
SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits
remarkable consistency in both structure and semantics. Project page at
https://hustvl.github.io/PixelHacker.
6814e35a19162d7749852c4b
Ziyang Xu
claimed_verified
2025-05-04T10:04:40.638Z
633d4630e1aec4b8b33ad5b8
https://cdn-avatars.huggingface.co/v1/production/uploads/633d4630e1aec4b8b33ad5b8/owFqnyCMW8yuo9VcjA1hx.jpeg
Ziyang Xu
user
Uyoung
6814e35a19162d7749852c4c
Kangsheng Duan
claimed_verified
2025-05-06T16:53:10.183Z
66d0ba2420e64fc408564bbb
/avatars/355566cfb6347750a5faab7c87fcb79c.svg
Anduin Dan
user
AnduinD
6814e35a19162d7749852c4d
Xiaolei Shen
admin_assigned
2025-05-05T07:32:16.185Z
627a34dac488a8ce15a2dc4a
/avatars/61aecef507dea6620fe5574493f83595.svg
ShenXiaolei
user
SmileTAT
6814e35a19162d7749852c4e
Zhifeng Ding
6814e35a19162d7749852c4f
Wenyu Liu
admin_assigned
2025-05-05T07:32:31.210Z
66c2e7fc934e2f07753542ac
/avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
LWY
user
wenyuliu
6814e35a19162d7749852c50
Xiaohu Ruan
6814e35a19162d7749852c51
Xiaoxin Chen
admin_assigned
2025-05-05T07:32:55.012Z
65389a669c474315d7425f96
/avatars/2fa3828ca489cfe1948129a0eccf264f.svg
chenxiaoxin
user
steelozazala
6814e35a19162d7749852c52
Xinggang Wang
admin_assigned
2025-05-05T07:33:01.396Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
PixelHacker: Image Inpainting with Structural and Semantic Consistency
Image inpainting is a fundamental research area between image editing and
image generation. Recent state-of-the-art (SOTA) methods have explored novel
attention mechanisms, lightweight architectures, and context-aware modeling,
demonstrating impressive performance. However, they often struggle with complex
structure (e.g., texture, shape, spatial relations) and semantics (e.g., color
consistency, object restoration, and logical correctness), leading to artifacts
and inappropriate generation. To address this challenge, we design a simple yet
effective inpainting paradigm called latent categories guidance, and further
propose a diffusion-based model named PixelHacker. Specifically, we first
construct a large dataset containing 14 million image-mask pairs by annotating
foreground and background (potential 116 and 21 categories, respectively).
Then, we encode potential foreground and background representations separately
through two fixed-size embeddings, and intermittently inject these features
into the denoising process via linear attention. Finally, by pre-training on
our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker.
Extensive experiments show that PixelHacker comprehensively outperforms the
SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits
remarkable consistency in both structure and semantics. Project page at
https://hustvl.github.io/PixelHacker.
PixelHacker, a diffusion-based model using latent categories guidance and linear attention, enhances image inpainting by improving structural and semantic consistency across various datasets.
latent categories guidance diffusion-based model PixelHacker linear attention image-mask pairs fixed-size embeddings denoising process Places2 CelebA-HQ FFHQ