UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models
Large language models, trained on extensive corpora, successfully unify
diverse linguistic tasks within a single generative framework. Inspired by
this, recent works like Large Vision Model (LVM) extend this paradigm to vision
by organizing tasks into sequential visual sentences, where visual prompts
serve as the context to guide outputs. However, such modeling requires
task-specific pre-training across modalities and sources, which is costly and
limits scalability to unseen tasks. Given that pre-trained video generation
models inherently capture temporal sequence dependencies, we explore a more
unified and scalable alternative: can a pre-trained video generation model
adapt to diverse image and video tasks? To answer this, we propose UniVid, a
framework that fine-tunes a video diffusion transformer to handle various
vision tasks without task-specific modifications. Tasks are represented as
visual sentences, where the context sequence defines both the task and the
expected output modality. We evaluate the generalization of UniVid from two
perspectives: (1) cross-modal inference with contexts composed of both images
and videos, extending beyond LVM's uni-modal setting; (2) cross-source tasks
from natural to annotated data, without multi-source pre-training. Despite
being trained solely on natural video data, UniVid generalizes well in both
settings. Notably, understanding and generation tasks can easily switch by
simply reversing the visual sentence order in this paradigm. These findings
highlight the potential of pre-trained video generation models to serve as a
scalable and unified foundation for vision modeling. Our code will be released
at https://github.com/CUC-MIPG/UniVid.
68d9e7890177a6054b013a25
Lan Chen
68d9e7890177a6054b013a26
Yuchao Gu
68d9e7890177a6054b013a27
Qi Mao
claimed_verified
2025-09-29T06:00:42.049Z
6388a7e98a5dbe2f3dc61faa
https://cdn-avatars.huggingface.co/v1/production/uploads/6388a7e98a5dbe2f3dc61faa/Zj8pR7BG_wOQjsYzosb5p.jpeg
Qi Mao
user
HelenMao
UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models
Large language models, trained on extensive corpora, successfully unify
diverse linguistic tasks within a single generative framework. Inspired by
this, recent works like Large Vision Model (LVM) extend this paradigm to vision
by organizing tasks into sequential visual sentences, where visual prompts
serve as the context to guide outputs. However, such modeling requires
task-specific pre-training across modalities and sources, which is costly and
limits scalability to unseen tasks. Given that pre-trained video generation
models inherently capture temporal sequence dependencies, we explore a more
unified and scalable alternative: can a pre-trained video generation model
adapt to diverse image and video tasks? To answer this, we propose UniVid, a
framework that fine-tunes a video diffusion transformer to handle various
vision tasks without task-specific modifications. Tasks are represented as
visual sentences, where the context sequence defines both the task and the
expected output modality. We evaluate the generalization of UniVid from two
perspectives: (1) cross-modal inference with contexts composed of both images
and videos, extending beyond LVM's uni-modal setting; (2) cross-source tasks
from natural to annotated data, without multi-source pre-training. Despite
being trained solely on natural video data, UniVid generalizes well in both
settings. Notably, understanding and generation tasks can easily switch by
simply reversing the visual sentence order in this paradigm. These findings
highlight the potential of pre-trained video generation models to serve as a
scalable and unified foundation for vision modeling. Our code will be released
at https://github.com/CUC-MIPG/UniVid.
A pre-trained video diffusion transformer, UniVid, is fine-tuned to handle diverse vision tasks without task-specific modifications, demonstrating cross-modal and cross-source generalization.
Large Vision Model visual sentences video diffusion transformer UniVid cross-modal inference cross-source tasks natural video data understanding and generation tasks