Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
Diffusion models, and their generalization, flow matching, have had a
remarkable impact on the field of media generation. Here, the conventional
approach is to learn the complex mapping from a simple source distribution of
Gaussian noise to the target media distribution. For cross-modal tasks such as
text-to-image generation, this same mapping from noise to image is learnt
whilst including a conditioning mechanism in the model. One key and thus far
relatively unexplored feature of flow matching is that, unlike Diffusion
models, they are not constrained for the source distribution to be noise.
Hence, in this paper, we propose a paradigm shift, and ask the question of
whether we can instead train flow matching models to learn a direct mapping
from the distribution of one modality to the distribution of another, thus
obviating the need for both the noise distribution and conditioning mechanism.
We present a general and simple framework, CrossFlow, for cross-modal flow
matching. We show the importance of applying Variational Encoders to the input
data, and introduce a method to enable Classifier-free guidance. Surprisingly,
for text-to-image, CrossFlow with a vanilla transformer without cross attention
slightly outperforms standard flow matching, and we show that it scales better
with training steps and model size, while also allowing for interesting latent
arithmetic which results in semantically meaningful edits in the output space.
To demonstrate the generalizability of our approach, we also show that
CrossFlow is on par with or outperforms the state-of-the-art for various
cross-modal / intra-modal mapping tasks, viz. image captioning, depth
estimation, and image super-resolution. We hope this paper contributes to
accelerating progress in cross-modal media generation.
6764dd4d469e108330dc958a
Qihao Liu
admin_assigned
2024-12-20T09:21:57.934Z
639f1e519f1f2baab2f00d22
https://cdn-avatars.huggingface.co/v1/production/uploads/639f1e519f1f2baab2f00d22/pFjd51WZuVZ3A11rItvmk.jpeg
Qihao Liu
user
QHL067
6764dd4d469e108330dc958b
Xi Yin
6764dd4d469e108330dc958c
Alan Yuille
6764dd4d469e108330dc958d
Andrew Brown
6764dd4d469e108330dc958e
Mannat Singh
admin_assigned
2024-12-20T09:22:34.696Z
650fece35877b1c077298b81
/avatars/be9c948ecf9c413ad4c8aa6598d3ba86.svg
Mannat Singh
user
mannatsingh
Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
Diffusion models, and their generalization, flow matching, have had a
remarkable impact on the field of media generation. Here, the conventional
approach is to learn the complex mapping from a simple source distribution of
Gaussian noise to the target media distribution. For cross-modal tasks such as
text-to-image generation, this same mapping from noise to image is learnt
whilst including a conditioning mechanism in the model. One key and thus far
relatively unexplored feature of flow matching is that, unlike Diffusion
models, they are not constrained for the source distribution to be noise.
Hence, in this paper, we propose a paradigm shift, and ask the question of
whether we can instead train flow matching models to learn a direct mapping
from the distribution of one modality to the distribution of another, thus
obviating the need for both the noise distribution and conditioning mechanism.
We present a general and simple framework, CrossFlow, for cross-modal flow
matching. We show the importance of applying Variational Encoders to the input
data, and introduce a method to enable Classifier-free guidance. Surprisingly,
for text-to-image, CrossFlow with a vanilla transformer without cross attention
slightly outperforms standard flow matching, and we show that it scales better
with training steps and model size, while also allowing for interesting latent
arithmetic which results in semantically meaningful edits in the output space.
To demonstrate the generalizability of our approach, we also show that
CrossFlow is on par with or outperforms the state-of-the-art for various
cross-modal / intra-modal mapping tasks, viz. image captioning, depth
estimation, and image super-resolution. We hope this paper contributes to
accelerating progress in cross-modal media generation.
CrossFlow proposes a direct mapping between modalities for cross-modal media generation using flow matching, outperforming standard methods in various tasks.
diffusion models flow matching Gaussian noise cross-modal tasks text-to-image generation noise distribution conditioning mechanism CrossFlow Variational Encoders Classifier-free guidance image captioning depth estimation image super-resolution