PhysMaster: Mastering Physical Representation for Video Generation via
  Reinforcement Learning
Video generation models nowadays are capable of generating visually realistic
videos, but often fail to adhere to physical laws, limiting their ability to
generate physically plausible videos and serve as ''world models''. To address
this issue, we propose PhysMaster, which captures physical knowledge as a
representation for guiding video generation models to enhance their
physics-awareness. Specifically, PhysMaster is based on the image-to-video task
where the model is expected to predict physically plausible dynamics from the
input image. Since the input image provides physical priors like relative
positions and potential interactions of objects in the scenario, we devise
PhysEncoder to encode physical information from it as an extra condition to
inject physical knowledge into the video generation process. The lack of proper
supervision on the model's physical performance beyond mere appearance
motivates PhysEncoder to apply reinforcement learning with human feedback to
physical representation learning, which leverages feedback from generation
models to optimize physical representations with Direct Preference Optimization
(DPO) in an end-to-end manner. PhysMaster provides a feasible solution for
improving physics-awareness of PhysEncoder and thus of video generation,
proving its ability on a simple proxy task and generalizability to wide-ranging
physical scenarios. This implies that our PhysMaster, which unifies solutions
for various physical processes via representation learning in the reinforcement
learning paradigm, can act as a generic and plug-in solution for physics-aware
video generation and broader applications.
68f04e9936f8b025381e18d2
Sihui Ji
claimed_verified
2025-10-16T06:39:18.148Z
64775b6ce5ddd7d2a16fadee
/avatars/19af37ffd7626ebc58b387a34a8f98d7.svg
Sihui Ji
user
zjuJish
68f04e9936f8b025381e18d3
Xi Chen
68f04e9936f8b025381e18d4
Xin Tao
68f04e9936f8b025381e18d5
Pengfei Wan
68f04e9936f8b025381e18d6
Hengshuang Zhao
PhysMaster: Mastering Physical Representation for Video Generation via
  Reinforcement Learning
Video generation models nowadays are capable of generating visually realistic
videos, but often fail to adhere to physical laws, limiting their ability to
generate physically plausible videos and serve as ''world models''. To address
this issue, we propose PhysMaster, which captures physical knowledge as a
representation for guiding video generation models to enhance their
physics-awareness. Specifically, PhysMaster is based on the image-to-video task
where the model is expected to predict physically plausible dynamics from the
input image. Since the input image provides physical priors like relative
positions and potential interactions of objects in the scenario, we devise
PhysEncoder to encode physical information from it as an extra condition to
inject physical knowledge into the video generation process. The lack of proper
supervision on the model's physical performance beyond mere appearance
motivates PhysEncoder to apply reinforcement learning with human feedback to
physical representation learning, which leverages feedback from generation
models to optimize physical representations with Direct Preference Optimization
(DPO) in an end-to-end manner. PhysMaster provides a feasible solution for
improving physics-awareness of PhysEncoder and thus of video generation,
proving its ability on a simple proxy task and generalizability to wide-ranging
physical scenarios. This implies that our PhysMaster, which unifies solutions
for various physical processes via representation learning in the reinforcement
learning paradigm, can act as a generic and plug-in solution for physics-aware
video generation and broader applications.
PhysMaster enhances video generation by integrating physical knowledge through PhysEncoder, using reinforcement learning and Direct Preference Optimization to improve physics-awareness.
PhysMaster PhysEncoder reinforcement learning Direct Preference Optimization physics-aware video generation