Generalizable Implicit Motion Modeling for Video Frame Interpolation
Motion modeling is critical in flow-based Video Frame Interpolation (VFI).
Existing paradigms either consider linear combinations of bidirectional flows
or directly predict bilateral flows for given timestamps without exploring
favorable motion priors, thus lacking the capability of effectively modeling
spatiotemporal dynamics in real-world videos. To address this limitation, in
this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel
and effective approach to motion modeling for VFI. Specifically, to enable GIMM
as an effective motion modeling paradigm, we design a motion encoding pipeline
to model spatiotemporal motion latent from bidirectional flows extracted from
pre-trained flow estimators, effectively representing input-specific motion
priors. Then, we implicitly predict arbitrary-timestep optical flows within two
adjacent input frames via an adaptive coordinate-based neural network, with
spatiotemporal coordinates and motion latent as inputs. Our GIMM can be
smoothly integrated with existing flow-based VFI works without further
modifications. We show that GIMM performs better than the current state of the
art on the VFI benchmarks.
6690905937eadb9c578773e4
Zujin Guo
claimed_verified
2024-07-15T09:00:54.351Z
62f4b57a9437c4d04fb5e6d6
/avatars/3681e4948fba1317c80bbbf2326a5fcd.svg
ZUJIN GUO
user
GSean
6690905937eadb9c578773e5
Wei Li
6690905937eadb9c578773e6
Chen Change Loy
Generalizable Implicit Motion Modeling for Video Frame Interpolation
Motion modeling is critical in flow-based Video Frame Interpolation (VFI).
Existing paradigms either consider linear combinations of bidirectional flows
or directly predict bilateral flows for given timestamps without exploring
favorable motion priors, thus lacking the capability of effectively modeling
spatiotemporal dynamics in real-world videos. To address this limitation, in
this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel
and effective approach to motion modeling for VFI. Specifically, to enable GIMM
as an effective motion modeling paradigm, we design a motion encoding pipeline
to model spatiotemporal motion latent from bidirectional flows extracted from
pre-trained flow estimators, effectively representing input-specific motion
priors. Then, we implicitly predict arbitrary-timestep optical flows within two
adjacent input frames via an adaptive coordinate-based neural network, with
spatiotemporal coordinates and motion latent as inputs. Our GIMM can be
smoothly integrated with existing flow-based VFI works without further
modifications. We show that GIMM performs better than the current state of the
art on the VFI benchmarks.
A new motion modeling approach, Generalizable Implicit Motion Modeling (GIMM), uses a motion encoding pipeline and an adaptive coordinate-based neural network to predict optical flows and improve Video Frame Interpolation.
flow-based Video Frame Interpolation (VFI) bidirectional flows Generalizable Implicit Motion Modeling (GIMM) motion encoding pipeline spatiotemporal motion latent adaptive coordinate-based neural network optical flows