Video2Game: Real-time, Interactive, Realistic and Browser-Compatible
  Environment from a Single Video
Creating high-quality and interactive virtual environments, such as games and
simulators, often involves complex and costly manual modeling processes. In
this paper, we present Video2Game, a novel approach that automatically converts
videos of real-world scenes into realistic and interactive game environments.
At the heart of our system are three core components:(i) a neural radiance
fields (NeRF) module that effectively captures the geometry and visual
appearance of the scene; (ii) a mesh module that distills the knowledge from
NeRF for faster rendering; and (iii) a physics module that models the
interactions and physical dynamics among the objects. By following the
carefully designed pipeline, one can construct an interactable and actionable
digital replica of the real world. We benchmark our system on both indoor and
large-scale outdoor scenes. We show that we can not only produce
highly-realistic renderings in real-time, but also build interactive games on
top.
661df62686e7bb7bede31a9a
Hongchi Xia
661df62686e7bb7bede31a9b
Zhi-Hao Lin
admin_assigned
2024-04-18T09:40:59.523Z
632a340c5f2ff1958c014191
/avatars/d51ccb2e2b3e9756dea3d06b6e62f438.svg
Chih-Hao Lin
user
chih-hao-lin
661df62686e7bb7bede31a9c
Wei-Chiu Ma
admin_assigned
2024-04-18T09:41:06.244Z
624fb0c8841b0a7f1b583e72
/avatars/a695a528817f57a98c946056d6f9b2f9.svg
Wei-Chiu Ma
user
weichium
661df62686e7bb7bede31a9d
Shenlong Wang
admin_assigned
2024-04-18T09:41:13.167Z
62fbe0aaa80632fbd47b86f5
/avatars/8175e3ac325745714f17dbe000ea955f.svg
Shenlong Wang
user
shenlong
Video2Game: Real-time, Interactive, Realistic and Browser-Compatible
  Environment from a Single Video
Creating high-quality and interactive virtual environments, such as games and
simulators, often involves complex and costly manual modeling processes. In
this paper, we present Video2Game, a novel approach that automatically converts
videos of real-world scenes into realistic and interactive game environments.
At the heart of our system are three core components:(i) a neural radiance
fields (NeRF) module that effectively captures the geometry and visual
appearance of the scene; (ii) a mesh module that distills the knowledge from
NeRF for faster rendering; and (iii) a physics module that models the
interactions and physical dynamics among the objects. By following the
carefully designed pipeline, one can construct an interactable and actionable
digital replica of the real world. We benchmark our system on both indoor and
large-scale outdoor scenes. We show that we can not only produce
highly-realistic renderings in real-time, but also build interactive games on
top.
Video2Game converts real-world scenes into realistic interactive environments by combining neural radiance fields, mesh distillation, and physics modeling.
neural radiance fields NeRF mesh distillation physics modeling interactive environments