Discrete Flow Matching
Despite Flow Matching and diffusion models having emerged as powerful
generative paradigms for continuous variables such as images and videos, their
application to high-dimensional discrete data, such as language, is still
limited. In this work, we present Discrete Flow Matching, a novel discrete flow
paradigm designed specifically for generating discrete data. Discrete Flow
Matching offers several key contributions: (i) it works with a general family
of probability paths interpolating between source and target distributions;
(ii) it allows for a generic formula for sampling from these probability paths
using learned posteriors such as the probability denoiser (x-prediction) and
noise-prediction (epsilon-prediction); (iii) practically, focusing on
specific probability paths defined with different schedulers considerably
improves generative perplexity compared to previous discrete diffusion and flow
models; and (iv) by scaling Discrete Flow Matching models up to 1.7B
parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1
and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of
generating high-quality discrete data in a non-autoregressive fashion,
significantly closing the gap between autoregressive models and discrete flow
models.
669fa3551d9ef5e557184206
Itai Gat
admin_assigned
2024-07-23T12:46:20.616Z
62a8fa984d933c74bf410c16
/avatars/73519deba3176be9c23d49f749aee5da.svg
Itai Gat
user
itaigat
669fa3551d9ef5e557184207
Tal Remez
admin_assigned
2024-07-23T12:46:26.570Z
647f1b66838ac3601fc762d1
/avatars/24aaaeb700690bc84ad0212ce4ae9bd4.svg
Tal Remez
user
TalRemez
669fa3551d9ef5e557184208
Neta Shaul
669fa3551d9ef5e557184209
Felix Kreuk
admin_assigned
2024-07-23T12:46:42.545Z
645d0a4cf4f49de5809de159
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/IOWMd17Iwls0dXsY1OWjK.jpeg
Felix Kreuk
user
felixkreuk
669fa3551d9ef5e55718420a
Ricky T. Q. Chen
669fa3551d9ef5e55718420b
Gabriel Synnaeve
admin_assigned
2024-07-23T12:47:07.756Z
630eac7931970d1cd4fbacf2
/avatars/b7ccbddfa745db854dc342be1327cd53.svg
Gabriel Synnaeve
user
gsynnaeve
669fa3551d9ef5e55718420c
Yossi Adi
admin_assigned
2024-07-23T12:47:13.577Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
669fa3551d9ef5e55718420d
Yaron Lipman
Discrete Flow Matching
Despite Flow Matching and diffusion models having emerged as powerful
generative paradigms for continuous variables such as images and videos, their
application to high-dimensional discrete data, such as language, is still
limited. In this work, we present Discrete Flow Matching, a novel discrete flow
paradigm designed specifically for generating discrete data. Discrete Flow
Matching offers several key contributions: (i) it works with a general family
of probability paths interpolating between source and target distributions;
(ii) it allows for a generic formula for sampling from these probability paths
using learned posteriors such as the probability denoiser (x-prediction) and
noise-prediction (epsilon-prediction); (iii) practically, focusing on
specific probability paths defined with different schedulers considerably
improves generative perplexity compared to previous discrete diffusion and flow
models; and (iv) by scaling Discrete Flow Matching models up to 1.7B
parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1
and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of
generating high-quality discrete data in a non-autoregressive fashion,
significantly closing the gap between autoregressive models and discrete flow
models.
Discrete Flow Matching is a novel generative model for discrete data that achieves high performance without an autoregressive approach, improving benchmarks like HumanEval and MBPP.
Flow Matching diffusion models discrete flow probability paths $x$-prediction $\epsilon$-prediction generative perplexity HumanEval 1-shot MBPP high-quality discrete data non-autoregressive models