Toward Guidance-Free AR Visual Generation via Condition Contrastive
  Alignment
Classifier-Free Guidance (CFG) is a critical technique for enhancing the
sample quality of visual generative models. However, in autoregressive (AR)
multi-modal generation, CFG introduces design inconsistencies between language
and visual content, contradicting the design philosophy of unifying different
modalities for visual AR. Motivated by language model alignment methods, we
propose Condition Contrastive Alignment (CCA) to facilitate
guidance-free AR visual generation with high performance and analyze its
theoretical connection with guided sampling methods. Unlike guidance methods
that alter the sampling process to achieve the ideal sampling distribution, CCA
directly fine-tunes pretrained models to fit the same distribution target.
Experimental results show that CCA can significantly enhance the guidance-free
performance of all tested models with just one epoch of fine-tuning (sim 1\%
of pretraining epochs) on the pretraining dataset, on par with guided sampling
methods. This largely removes the need for guided sampling in AR visual
generation and cuts the sampling cost by half. Moreover, by adjusting training
parameters, CCA can achieve trade-offs between sample diversity and fidelity
similar to CFG. This experimentally confirms the strong theoretical connection
between language-targeted alignment and visual-targeted guidance methods,
unifying two previously independent research fields. Code and model weights:
https://github.com/thu-ml/CCA.
6711f37b3eea86736062288b
Huayu Chen
claimed_verified
2024-10-18T08:09:41.392Z
65571135bfb62d747abc8129
/avatars/5f4542daa34597f17e6280b9cce18c91.svg
Hugging
user
ChenDRAG
6711f37b3eea86736062288c
Hang Su
6711f37b3eea86736062288d
Peize Sun
admin_assigned
2024-10-18T16:40:52.212Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
6711f37b3eea86736062288e
Jun Zhu
Toward Guidance-Free AR Visual Generation via Condition Contrastive
  Alignment
Classifier-Free Guidance (CFG) is a critical technique for enhancing the
sample quality of visual generative models. However, in autoregressive (AR)
multi-modal generation, CFG introduces design inconsistencies between language
and visual content, contradicting the design philosophy of unifying different
modalities for visual AR. Motivated by language model alignment methods, we
propose Condition Contrastive Alignment (CCA) to facilitate
guidance-free AR visual generation with high performance and analyze its
theoretical connection with guided sampling methods. Unlike guidance methods
that alter the sampling process to achieve the ideal sampling distribution, CCA
directly fine-tunes pretrained models to fit the same distribution target.
Experimental results show that CCA can significantly enhance the guidance-free
performance of all tested models with just one epoch of fine-tuning (sim 1\%
of pretraining epochs) on the pretraining dataset, on par with guided sampling
methods. This largely removes the need for guided sampling in AR visual
generation and cuts the sampling cost by half. Moreover, by adjusting training
parameters, CCA can achieve trade-offs between sample diversity and fidelity
similar to CFG. This experimentally confirms the strong theoretical connection
between language-targeted alignment and visual-targeted guidance methods,
unifying two previously independent research fields. Code and model weights:
https://github.com/thu-ml/CCA.
Condition Contrastive Alignment (CCA) enhances autoregressive visual generation without guided sampling by fine-tuning pretrained models, achieving high performance and balancing sample diversity and fidelity.
Classifier-Free Guidance (CFG) autoregressive (AR) multi-modal generation Condition Contrastive Alignment (CCA) guided sampling methods pretraining dataset sampling cost sample diversity sample fidelity