MATRIX: Mask Track Alignment for Interaction-aware Video Generation
Video DiTs have advanced video generation, yet they still struggle to model
multi-instance or subject-object interactions. This raises a key question: How
do these models internally represent interactions? To answer this, we curate
MATRIX-11K, a video dataset with interaction-aware captions and multi-instance
mask tracks. Using this dataset, we conduct a systematic analysis that
formalizes two perspectives of video DiTs: semantic grounding, via
video-to-text attention, which evaluates whether noun and verb tokens capture
instances and their relations; and semantic propagation, via video-to-video
attention, which assesses whether instance bindings persist across frames. We
find both effects concentrate in a small subset of interaction-dominant layers.
Motivated by this, we introduce MATRIX, a simple and effective regularization
that aligns attention in specific layers of video DiTs with multi-instance mask
tracks from the MATRIX-11K dataset, enhancing both grounding and propagation.
We further propose InterGenEval, an evaluation protocol for interaction-aware
video generation. In experiments, MATRIX improves both interaction fidelity and
semantic alignment while reducing drift and hallucination. Extensive ablations
validate our design choices. Codes and weights will be released.
68e71a787ae125f9582e6969
Siyoon Jin
claimed_verified
2025-10-09T12:48:49.499Z
67e3a3cc0c2f0d766d401bdb
/avatars/0de4c3b11295505ec9d3626e65302cbd.svg
Siyoon Jin
user
clwm515
68e71a787ae125f9582e696a
Seongchan Kim
claimed_verified
2025-10-09T06:50:18.453Z
637c49ec9c470afa3880b137
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pdcMPz8N6vQM1tc8IA1lV.png
Seongchan Kim
user
Seongchan
68e71a787ae125f9582e696b
Dahyun Chung
68e71a787ae125f9582e696c
Jaeho Lee
68e71a787ae125f9582e696d
Hyunwook Choi
68e71a787ae125f9582e696e
Jisu Nam
68e71a787ae125f9582e696f
Jiyoung Kim
68e71a787ae125f9582e6970
Seungryong Kim
MATRIX: Mask Track Alignment for Interaction-aware Video Generation
Video DiTs have advanced video generation, yet they still struggle to model
multi-instance or subject-object interactions. This raises a key question: How
do these models internally represent interactions? To answer this, we curate
MATRIX-11K, a video dataset with interaction-aware captions and multi-instance
mask tracks. Using this dataset, we conduct a systematic analysis that
formalizes two perspectives of video DiTs: semantic grounding, via
video-to-text attention, which evaluates whether noun and verb tokens capture
instances and their relations; and semantic propagation, via video-to-video
attention, which assesses whether instance bindings persist across frames. We
find both effects concentrate in a small subset of interaction-dominant layers.
Motivated by this, we introduce MATRIX, a simple and effective regularization
that aligns attention in specific layers of video DiTs with multi-instance mask
tracks from the MATRIX-11K dataset, enhancing both grounding and propagation.
We further propose InterGenEval, an evaluation protocol for interaction-aware
video generation. In experiments, MATRIX improves both interaction fidelity and
semantic alignment while reducing drift and hallucination. Extensive ablations
validate our design choices. Codes and weights will be released.
MATRIX-11K dataset and MATRIX regularization enhance interaction fidelity and semantic alignment in video DiTs by aligning attention with multi-instance mask tracks.
video DiTs interaction-aware captions multi-instance mask tracks video-to-text attention video-to-video attention semantic grounding semantic propagation MATRIX regularization InterGenEval interaction fidelity semantic alignment drift hallucination