Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
Recent video editing methods achieve attractive results in style transfer or
appearance modification. However, editing the structural content of 3D scenes
in videos remains challenging, particularly when dealing with significant
viewpoint changes, such as large camera rotations or zooms. Key challenges
include generating novel view content that remains consistent with the original
video, preserving unedited regions, and translating sparse 2D inputs into
realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a
sketch-based 3D-aware video editing method to enable detailed local
manipulation of videos with significant viewpoint changes. To solve the
challenge posed by sparse inputs, we employ image editing methods to generate
edited results for the first frame, which are then propagated to the remaining
frames of the video. We utilize sketching as an interaction tool for precise
geometry control, while other mask-based image editing methods are also
supported. To handle viewpoint changes, we perform a detailed analysis and
manipulation of the 3D information in the video. Specifically, we utilize a
dense stereo method to estimate a point cloud and the camera parameters of the
input video. We then propose a point cloud editing approach that uses depth
maps to represent the 3D geometry of newly edited components, aligning them
effectively with the original 3D scene. To seamlessly merge the newly edited
content with the original video while preserving the features of unedited
regions, we introduce a 3D-aware mask propagation strategy and employ a video
diffusion model to produce realistic edited videos. Extensive experiments
demonstrate the superiority of Sketch3DVE in video editing. Homepage and code:
http://http://geometrylearning.com/Sketch3DVE/
68a5a31dc4b3ea17d2d2cf4d
Feng-Lin Liu
admin_assigned
2025-08-25T20:33:36.346Z
6424538b9f9e65b42389920e
/avatars/9b912e2af9eebe9a481181f006765059.svg
Feng-Lin Liu
user
Okrin
68a5a31dc4b3ea17d2d2cf4e
Shi-Yang Li
68a5a31dc4b3ea17d2d2cf4f
Yan-Pei Cao
68a5a31dc4b3ea17d2d2cf50
Hongbo Fu
68a5a31dc4b3ea17d2d2cf51
Lin Gao
Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
Recent video editing methods achieve attractive results in style transfer or
appearance modification. However, editing the structural content of 3D scenes
in videos remains challenging, particularly when dealing with significant
viewpoint changes, such as large camera rotations or zooms. Key challenges
include generating novel view content that remains consistent with the original
video, preserving unedited regions, and translating sparse 2D inputs into
realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a
sketch-based 3D-aware video editing method to enable detailed local
manipulation of videos with significant viewpoint changes. To solve the
challenge posed by sparse inputs, we employ image editing methods to generate
edited results for the first frame, which are then propagated to the remaining
frames of the video. We utilize sketching as an interaction tool for precise
geometry control, while other mask-based image editing methods are also
supported. To handle viewpoint changes, we perform a detailed analysis and
manipulation of the 3D information in the video. Specifically, we utilize a
dense stereo method to estimate a point cloud and the camera parameters of the
input video. We then propose a point cloud editing approach that uses depth
maps to represent the 3D geometry of newly edited components, aligning them
effectively with the original 3D scene. To seamlessly merge the newly edited
content with the original video while preserving the features of unedited
regions, we introduce a 3D-aware mask propagation strategy and employ a video
diffusion model to produce realistic edited videos. Extensive experiments
demonstrate the superiority of Sketch3DVE in video editing. Homepage and code:
http://http://geometrylearning.com/Sketch3DVE/
Sketch3DVE is a sketch-based 3D-aware video editing method that uses dense stereo estimation, point cloud editing, and a video diffusion model to handle sparse inputs and viewpoint changes in video editing.
dense stereo method point cloud camera parameters point cloud editing depth maps 3D-aware mask propagation video diffusion model