Drag View: Generalizable Novel View Synthesis with Unposed Imagery
We introduce DragView, a novel and interactive framework for generating novel
views of unseen scenes. DragView initializes the new view from a single source
image, and the rendering is supported by a sparse set of unposed multi-view
images, all seamlessly executed within a single feed-forward pass. Our approach
begins with users dragging a source view through a local relative coordinate
system. Pixel-aligned features are obtained by projecting the sampled 3D points
along the target ray onto the source view. We then incorporate a view-dependent
modulation layer to effectively handle occlusion during the projection.
Additionally, we broaden the epipolar attention mechanism to encompass all
source pixels, facilitating the aggregation of initialized coordinate-aligned
point features from other unposed views. Finally, we employ another transformer
to decode ray features into final pixel intensities. Crucially, our framework
does not rely on either 2D prior models or the explicit estimation of camera
poses. During testing, DragView showcases the capability to generalize to new
scenes unseen during training, also utilizing only unposed support images,
enabling the generation of photo-realistic new views characterized by flexible
camera trajectories. In our experiments, we conduct a comprehensive comparison
of the performance of DragView with recent scene representation networks
operating under pose-free conditions, as well as with generalizable NeRFs
subject to noisy test camera poses. DragView consistently demonstrates its
superior performance in view synthesis quality, while also being more
user-friendly. Project page: https://zhiwenfan.github.io/DragView/.
651f6ab9630a846432fbdcc2
Zhiwen Fan
651f6ab9630a846432fbdcc3
Panwang Pan
claimed_verified
2023-11-07T11:20:55.777Z
654866e8cd0a5621395f8287
https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
Panwang Pan
user
paulpanwang
651f6ab9630a846432fbdcc4
Peihao Wang
admin_assigned
2023-10-10T08:38:34.953Z
62c8f1773e28b8ee4cddca09
/avatars/9edbcc84e4ab2b335011daab8acfdbb8.svg
Peihao Wang
user
peihaowang
651f6ab9630a846432fbdcc5
Yifan Jiang
651f6ab9630a846432fbdcc6
Hanwen Jiang
651f6ab9630a846432fbdcc7
Dejia Xu
admin_assigned
2023-10-10T08:40:06.828Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
651f6ab9630a846432fbdcc8
Zehao Zhu
admin_assigned
2023-10-10T08:40:19.933Z
63611b5b6bd72c97d0fe5dc8
/avatars/8905d4c0a930c7ba6e057c8b01f22a2e.svg
Zehao Zhu
user
zzhzzh
651f6ab9630a846432fbdcc9
Dilin Wang
651f6ab9630a846432fbdcca
Zhangyang Wang
Drag View: Generalizable Novel View Synthesis with Unposed Imagery
We introduce DragView, a novel and interactive framework for generating novel
views of unseen scenes. DragView initializes the new view from a single source
image, and the rendering is supported by a sparse set of unposed multi-view
images, all seamlessly executed within a single feed-forward pass. Our approach
begins with users dragging a source view through a local relative coordinate
system. Pixel-aligned features are obtained by projecting the sampled 3D points
along the target ray onto the source view. We then incorporate a view-dependent
modulation layer to effectively handle occlusion during the projection.
Additionally, we broaden the epipolar attention mechanism to encompass all
source pixels, facilitating the aggregation of initialized coordinate-aligned
point features from other unposed views. Finally, we employ another transformer
to decode ray features into final pixel intensities. Crucially, our framework
does not rely on either 2D prior models or the explicit estimation of camera
poses. During testing, DragView showcases the capability to generalize to new
scenes unseen during training, also utilizing only unposed support images,
enabling the generation of photo-realistic new views characterized by flexible
camera trajectories. In our experiments, we conduct a comprehensive comparison
of the performance of DragView with recent scene representation networks
operating under pose-free conditions, as well as with generalizable NeRFs
subject to noisy test camera poses. DragView consistently demonstrates its
superior performance in view synthesis quality, while also being more
user-friendly. Project page: https://zhiwenfan.github.io/DragView/.
DragView is an interactive framework that generates novel views of unseen scenes using a single source image and sparse unposed multi-view images, utilizing an epipolar attention mechanism and transformer-based ray decoding without estimating camera poses.
interactive framework DragView view synthesis source image unposed multi-view images feed-forward pass pixel-aligned features view-dependent modulation epipolar attention mechanism transformer ray features photo-realistic views scene representation networks pose-free conditions NeRF