AMO Sampler: Enhancing Text Rendering with Overshooting
Achieving precise alignment between textual instructions and generated images
in text-to-image generation is a significant challenge, particularly in
rendering written text within images. Sate-of-the-art models like Stable
Diffusion 3 (SD3), Flux, and AuraFlow still struggle with accurate text
depiction, resulting in misspelled or inconsistent text. We introduce a
training-free method with minimal computational overhead that significantly
enhances text rendering quality. Specifically, we introduce an overshooting
sampler for pretrained rectified flow (RF) models, by alternating between
over-simulating the learned ordinary differential equation (ODE) and
reintroducing noise. Compared to the Euler sampler, the overshooting sampler
effectively introduces an extra Langevin dynamics term that can help correct
the compounding error from successive Euler steps and therefore improve the
text rendering. However, when the overshooting strength is high, we observe
over-smoothing artifacts on the generated images. To address this issue, we
propose an Attention Modulated Overshooting sampler (AMO), which adaptively
controls the strength of overshooting for each image patch according to their
attention score with the text content. AMO demonstrates a 32.3% and 35.9%
improvement in text rendering accuracy on SD3 and Flux without compromising
overall image quality or increasing inference cost.
674fac3ae3ba9e6539cda7c0
Xixi Hu
674fac3ae3ba9e6539cda7c1
Keyang Xu
674fac3ae3ba9e6539cda7c2
Bo Liu
claimed_verified
2024-12-06T09:09:50.276Z
64375a5ab8ea86e50329d2da
/avatars/b3dcef9e2e0234156a22d699a30b6f1f.svg
Bo Liu
user
cranial-xix
674fac3ae3ba9e6539cda7c3
Qiang Liu
674fac3ae3ba9e6539cda7c4
Hongliang Fei
AMO Sampler: Enhancing Text Rendering with Overshooting
Achieving precise alignment between textual instructions and generated images
in text-to-image generation is a significant challenge, particularly in
rendering written text within images. Sate-of-the-art models like Stable
Diffusion 3 (SD3), Flux, and AuraFlow still struggle with accurate text
depiction, resulting in misspelled or inconsistent text. We introduce a
training-free method with minimal computational overhead that significantly
enhances text rendering quality. Specifically, we introduce an overshooting
sampler for pretrained rectified flow (RF) models, by alternating between
over-simulating the learned ordinary differential equation (ODE) and
reintroducing noise. Compared to the Euler sampler, the overshooting sampler
effectively introduces an extra Langevin dynamics term that can help correct
the compounding error from successive Euler steps and therefore improve the
text rendering. However, when the overshooting strength is high, we observe
over-smoothing artifacts on the generated images. To address this issue, we
propose an Attention Modulated Overshooting sampler (AMO), which adaptively
controls the strength of overshooting for each image patch according to their
attention score with the text content. AMO demonstrates a 32.3% and 35.9%
improvement in text rendering accuracy on SD3 and Flux without compromising
overall image quality or increasing inference cost.
A novel training-free method using an overshooting sampler enhances text rendering quality in text-to-image generation without increasing computational overhead.
Stable Diffusion 3 Flux AuraFlow pretrained rectified flow ordinary differential equation Euler sampler Langevin dynamics Attention Modulated Overshooting sampler text rendering accuracy inference cost