AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning
  in 4D Scenes
Recent Text-to-Video (T2V) models have demonstrated powerful capability in
visual simulation of real-world geometry and physical laws, indicating its
potential as implicit world models. Inspired by this, we explore the
feasibility of leveraging the video generation prior for viewpoint planning
from given 4D scenes, since videos internally accompany dynamic scenes with
natural viewpoints. To this end, we propose a two-stage paradigm to adapt
pre-trained T2V models for viewpoint prediction, in a compatible manner. First,
we inject the 4D scene representation into the pre-trained T2V model via an
adaptive learning branch, where the 4D scene is viewpoint-agnostic and the
conditional generated video embeds the viewpoints visually. Then, we formulate
viewpoint extraction as a hybrid-condition guided camera extrinsic denoising
process. Specifically, a camera extrinsic diffusion branch is further
introduced onto the pre-trained T2V model, by taking the generated video and 4D
scene as input. Experimental results show the superiority of our proposed
method over existing competitors, and ablation studies validate the
effectiveness of our key technical designs. To some extent, this work proves
the potential of video generation models toward 4D interaction in real world.
68edaa98de1fee572713a6d8
Yu Li
68edaa98de1fee572713a6d9
Menghan Xia
68edaa98de1fee572713a6da
Gongye Liu
68edaa98de1fee572713a6db
Jianhong Bai
68edaa98de1fee572713a6dc
Xintao Wang
68edaa98de1fee572713a6dd
Conglang Zhang
68edaa98de1fee572713a6de
Yuxuan Lin
68edaa98de1fee572713a6df
Ruihang Chu
68edaa98de1fee572713a6e0
Pengfei Wan
68edaa98de1fee572713a6e1
Yujiu Yang
AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning
  in 4D Scenes
Recent Text-to-Video (T2V) models have demonstrated powerful capability in
visual simulation of real-world geometry and physical laws, indicating its
potential as implicit world models. Inspired by this, we explore the
feasibility of leveraging the video generation prior for viewpoint planning
from given 4D scenes, since videos internally accompany dynamic scenes with
natural viewpoints. To this end, we propose a two-stage paradigm to adapt
pre-trained T2V models for viewpoint prediction, in a compatible manner. First,
we inject the 4D scene representation into the pre-trained T2V model via an
adaptive learning branch, where the 4D scene is viewpoint-agnostic and the
conditional generated video embeds the viewpoints visually. Then, we formulate
viewpoint extraction as a hybrid-condition guided camera extrinsic denoising
process. Specifically, a camera extrinsic diffusion branch is further
introduced onto the pre-trained T2V model, by taking the generated video and 4D
scene as input. Experimental results show the superiority of our proposed
method over existing competitors, and ablation studies validate the
effectiveness of our key technical designs. To some extent, this work proves
the potential of video generation models toward 4D interaction in real world.
A two-stage paradigm adapts pre-trained Text-to-Video models for viewpoint prediction in 4D scenes by integrating an adaptive learning branch and a camera extrinsic diffusion branch.
Text-to-Video models viewpoint planning 4D scenes adaptive learning branch viewpoint extraction hybrid-condition guided camera extrinsic denoising camera extrinsic diffusion branch