LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer
Recent advances in multimodal foundation models unifying image understanding
and generation have opened exciting avenues for tackling a wide range of
vision-language tasks within a single framework. Despite progress, existing
unified models typically require extensive pretraining and struggle to achieve
the same level of performance compared to models dedicated to each task.
Additionally, many of these models suffer from slow image generation speeds,
limiting their practical deployment in real-time or resource-constrained
settings. In this work, we propose Layerwise Timestep-Expert Flow-based
Transformer (LaTtE-Flow), a novel and efficient architecture that unifies image
understanding and generation within a single multimodal model. LaTtE-Flow
builds upon powerful pretrained Vision-Language Models (VLMs) to inherit strong
multimodal understanding capabilities, and extends them with a novel Layerwise
Timestep Experts flow-based architecture for efficient image generation.
LaTtE-Flow distributes the flow-matching process across specialized groups of
Transformer layers, each responsible for a distinct subset of timesteps. This
design significantly improves sampling efficiency by activating only a small
subset of layers at each sampling timestep. To further enhance performance, we
propose a Timestep-Conditioned Residual Attention mechanism for efficient
information reuse across layers. Experiments demonstrate that LaTtE-Flow
achieves strong performance on multimodal understanding tasks, while achieving
competitive image generation quality with around 6x faster inference speed
compared to recent unified multimodal models.
684b98083b733ba333687040
Ying Shen
684b98083b733ba333687041
Zhiyang Xu
claimed_verified
2025-06-13T07:39:29.999Z
665a443ce0bd04136ed3f163
/avatars/ffd910be7722713377e57b059268d2a4.svg
Zhiyang Xu
user
zhiyang1
684b98083b733ba333687042
Jiuhai Chen
684b98083b733ba333687043
Shizhe Diao
684b98083b733ba333687044
Jiaxin Zhang
684b98083b733ba333687045
Yuguang Yao
684b98083b733ba333687046
Joy Rimchala
684b98083b733ba333687047
Ismini Lourentzou
claimed_verified
2025-07-05T07:54:11.257Z
678ac3b31cbaa0b4bc295885
/avatars/1244fc1b305c9c6383df9bb5e4707347.svg
Ismini Lourentzou
user
isminoula
684b98083b733ba333687048
Lifu Huang
LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer
Recent advances in multimodal foundation models unifying image understanding
and generation have opened exciting avenues for tackling a wide range of
vision-language tasks within a single framework. Despite progress, existing
unified models typically require extensive pretraining and struggle to achieve
the same level of performance compared to models dedicated to each task.
Additionally, many of these models suffer from slow image generation speeds,
limiting their practical deployment in real-time or resource-constrained
settings. In this work, we propose Layerwise Timestep-Expert Flow-based
Transformer (LaTtE-Flow), a novel and efficient architecture that unifies image
understanding and generation within a single multimodal model. LaTtE-Flow
builds upon powerful pretrained Vision-Language Models (VLMs) to inherit strong
multimodal understanding capabilities, and extends them with a novel Layerwise
Timestep Experts flow-based architecture for efficient image generation.
LaTtE-Flow distributes the flow-matching process across specialized groups of
Transformer layers, each responsible for a distinct subset of timesteps. This
design significantly improves sampling efficiency by activating only a small
subset of layers at each sampling timestep. To further enhance performance, we
propose a Timestep-Conditioned Residual Attention mechanism for efficient
information reuse across layers. Experiments demonstrate that LaTtE-Flow
achieves strong performance on multimodal understanding tasks, while achieving
competitive image generation quality with around 6x faster inference speed
compared to recent unified multimodal models.
LaTtE-Flow, a new architecture, unifies image understanding and generation with high performance and faster inference by using a Layerwise Timestep Experts flow-based Transformer and Timestep-Conditioned Residual Attention mechanism.
Vision-Language Models Layerwise Timestep Experts flow-based architecture Transformer layers flow-matching process Timestep-Conditioned Residual Attention mechanism