What If : Understanding Motion Through Sparse Interactions
Understanding the dynamics of a physical scene involves reasoning about the
diverse ways it can potentially change, especially as a result of local
interactions. We present the Flow Poke Transformer (FPT), a novel framework for
directly predicting the distribution of local motion, conditioned on sparse
interactions termed "pokes". Unlike traditional methods that typically only
enable dense sampling of a single realization of scene dynamics, FPT provides
an interpretable directly accessible representation of multi-modal scene
motion, its dependency on physical interactions and the inherent uncertainties
of scene dynamics. We also evaluate our model on several downstream tasks to
enable comparisons with prior methods and highlight the flexibility of our
approach. On dense face motion generation, our generic pre-trained model
surpasses specialized baselines. FPT can be fine-tuned in strongly
out-of-distribution tasks such as synthetic datasets to enable significant
improvements over in-domain methods in articulated object motion estimation.
Additionally, predicting explicit motion distributions directly enables our
method to achieve competitive performance on tasks like moving part
segmentation from pokes which further demonstrates the versatility of our FPT.
Code and models are publicly available at
https://compvis.github.io/flow-poke-transformer.
68ef0228486b78128f0e33d7
Stefan Andreas Baumann
claimed_verified
2025-10-15T03:12:51.560Z
6471c12a0c2b5fdaf1f07c45
/avatars/6783067212d24d1e716a8b4c64df61b4.svg
Stefan Baumann
user
stefan-baumann
68ef0228486b78128f0e33d8
Nick Stracke
68ef0228486b78128f0e33d9
Timy Phan
68ef0228486b78128f0e33da
Björn Ommer
What If : Understanding Motion Through Sparse Interactions
Understanding the dynamics of a physical scene involves reasoning about the
diverse ways it can potentially change, especially as a result of local
interactions. We present the Flow Poke Transformer (FPT), a novel framework for
directly predicting the distribution of local motion, conditioned on sparse
interactions termed "pokes". Unlike traditional methods that typically only
enable dense sampling of a single realization of scene dynamics, FPT provides
an interpretable directly accessible representation of multi-modal scene
motion, its dependency on physical interactions and the inherent uncertainties
of scene dynamics. We also evaluate our model on several downstream tasks to
enable comparisons with prior methods and highlight the flexibility of our
approach. On dense face motion generation, our generic pre-trained model
surpasses specialized baselines. FPT can be fine-tuned in strongly
out-of-distribution tasks such as synthetic datasets to enable significant
improvements over in-domain methods in articulated object motion estimation.
Additionally, predicting explicit motion distributions directly enables our
method to achieve competitive performance on tasks like moving part
segmentation from pokes which further demonstrates the versatility of our FPT.
Code and models are publicly available at
https://compvis.github.io/flow-poke-transformer.