Discrete Diffusion Models with MLLMs for Unified Medical Multimodal
  Generation
Recent advances in generative medical models are constrained by
modality-specific scenarios that hinder the integration of complementary
evidence from imaging, pathology, and clinical notes. This fragmentation limits
their evolution into foundation models that can learn and reason across the
full spectrum of biomedical data. We propose MeDiM, the first medical discrete
diffusion model that learns shared distributions across modalities without
modality-specific components. MeDiM unifies multiple generative tasks:
translating between images and text, and jointly producing image-report pairs
across domains in response to prompts. Built on a discrete diffusion framework,
MeDiM bridges vision and language representations through a shared
probabilistic space. To enable unified and flexible medical generation, we
employ a multimodal large language model (MLLM) as the diffusion backbone,
leveraging its prior knowledge and cross-modal reasoning. Two key designs are
introduced: (1) removing the causal attention mask for bidirectional context,
and (2) injecting continuous timestep embeddings for diffusion awareness.
Experiments demonstrate high-fidelity medical generation (FID 16.60 on
MIMIC-CXR and FID 24.19 on PathGen) and accurate report generation (METEOR
0.2650 and 0.2580). Jointly generated image-report pairs further enhance
downstream performance (plus6.43 percent BLEU-1, plus18.57 percent BLEU-2,
plus31.58 percent BLEU-3, plus4.80 percent METEOR), showing that MeDiM supports
coherent and clinically grounded multimodal outputs.
68e5d525975ac4c405ef2118
Jiawei Mao
68e5d525975ac4c405ef2119
Yuhan Wang
68e5d525975ac4c405ef211a
Lifeng Chen
68e5d525975ac4c405ef211b
Can Zhao
68e5d525975ac4c405ef211c
Yucheng Tang
68e5d525975ac4c405ef211d
Dong Yang
68e5d525975ac4c405ef211e
Liangqiong Qu
68e5d525975ac4c405ef211f
Daguang Xu
68e5d525975ac4c405ef2120
Yuyin Zhou
Discrete Diffusion Models with MLLMs for Unified Medical Multimodal
  Generation
Recent advances in generative medical models are constrained by
modality-specific scenarios that hinder the integration of complementary
evidence from imaging, pathology, and clinical notes. This fragmentation limits
their evolution into foundation models that can learn and reason across the
full spectrum of biomedical data. We propose MeDiM, the first medical discrete
diffusion model that learns shared distributions across modalities without
modality-specific components. MeDiM unifies multiple generative tasks:
translating between images and text, and jointly producing image-report pairs
across domains in response to prompts. Built on a discrete diffusion framework,
MeDiM bridges vision and language representations through a shared
probabilistic space. To enable unified and flexible medical generation, we
employ a multimodal large language model (MLLM) as the diffusion backbone,
leveraging its prior knowledge and cross-modal reasoning. Two key designs are
introduced: (1) removing the causal attention mask for bidirectional context,
and (2) injecting continuous timestep embeddings for diffusion awareness.
Experiments demonstrate high-fidelity medical generation (FID 16.60 on
MIMIC-CXR and FID 24.19 on PathGen) and accurate report generation (METEOR
0.2650 and 0.2580). Jointly generated image-report pairs further enhance
downstream performance (plus6.43 percent BLEU-1, plus18.57 percent BLEU-2,
plus31.58 percent BLEU-3, plus4.80 percent METEOR), showing that MeDiM supports
coherent and clinically grounded multimodal outputs.
MeDiM, a medical discrete diffusion model, integrates multimodal biomedical data by learning shared distributions across images, text, and clinical notes, achieving high-fidelity generation and enhanced downstream performance.
discrete diffusion model multimodal large language model bidirectional context continuous timestep embeddings FID METEOR BLEU