Paris: A Decentralized Trained Open-Weight Diffusion Model
We present Paris, the first publicly released diffusion model pre-trained
entirely through decentralized computation. Paris demonstrates that
high-quality text-to-image generation can be achieved without centrally
coordinated infrastructure. Paris is open for research and commercial use.
Paris required implementing our Distributed Diffusion Training framework from
scratch. The model consists of 8 expert diffusion models (129M-605M parameters
each) trained in complete isolation with no gradient, parameter, or
intermediate activation synchronization. Rather than requiring synchronized
gradient updates across thousands of GPUs, we partition data into semantically
coherent clusters where each expert independently optimizes its subset while
collectively approximating the full distribution. A lightweight transformer
router dynamically selects appropriate experts at inference, achieving
generation quality comparable to centrally coordinated baselines. Eliminating
synchronization enables training on heterogeneous hardware without specialized
interconnects. Empirical validation confirms that Paris's decentralized
training maintains generation quality while removing the dedicated GPU cluster
requirement for large-scale diffusion models. Paris achieves this using
14times less training data and 16times less compute than the prior
decentralized baseline.
68e4735ce4e093a7044e4cd1
Zhiying Jiang
68e4735ce4e093a7044e4cd2
Raihan Seraj
68e4735ce4e093a7044e4cd3
Marcos Villagra
68e4735ce4e093a7044e4cd4
Bidhan Roy
Paris: A Decentralized Trained Open-Weight Diffusion Model
We present Paris, the first publicly released diffusion model pre-trained
entirely through decentralized computation. Paris demonstrates that
high-quality text-to-image generation can be achieved without centrally
coordinated infrastructure. Paris is open for research and commercial use.
Paris required implementing our Distributed Diffusion Training framework from
scratch. The model consists of 8 expert diffusion models (129M-605M parameters
each) trained in complete isolation with no gradient, parameter, or
intermediate activation synchronization. Rather than requiring synchronized
gradient updates across thousands of GPUs, we partition data into semantically
coherent clusters where each expert independently optimizes its subset while
collectively approximating the full distribution. A lightweight transformer
router dynamically selects appropriate experts at inference, achieving
generation quality comparable to centrally coordinated baselines. Eliminating
synchronization enables training on heterogeneous hardware without specialized
interconnects. Empirical validation confirms that Paris's decentralized
training maintains generation quality while removing the dedicated GPU cluster
requirement for large-scale diffusion models. Paris achieves this using
14times less training data and 16times less compute than the prior
decentralized baseline.
Paris, a decentralized diffusion model, achieves high-quality text-to-image generation without centralized infrastructure using a distributed training framework and a transformer router.
diffusion model decentralized computation Distributed Diffusion Training expert diffusion models parameter synchronization gradient updates semantically coherent clusters lightweight transformer router heterogeneous hardware decentralized training