Holistic Order Prediction in Natural Scenes
Pierre Musacchio, Hyunmin Lee, Jaesik Park
Even in controlled settings, understanding instance-wise geometries is a challenging task for a wide range of visual models. Although expert systems exist, modern arts still rely on expensive input formats (category labels, binary segmentation masks) and inference costs (quadratic amount of forward passes). We mitigate these limitations by proposing InstaFormer, a network capable of holistic order prediction. That is, solely given an input RGB image, InstaFormer returns the full segmentation masks along with occlusion and depth orderings for all the instances in the scene in a single forward pass. At its core, InstaFormer relies on interactions between object queries and latent mask descriptors that semantically represent the same objects while carrying complementary information. We comprehensively benchmark and ablate our approach to highlight its effectiveness. The link to our repository will be shared upon publication.