Projective Positional Encoding for Multiview Transformers
Ruilong Li, Brent Yi, Junchen Liu, Hang Gao, Yi Ma, Angjoo Kanazawa
As transformers become increasingly adopted for multi-view 3D perception tasks, a key challenge lies in effectively conditioning these models on camera parameters that ground visual observations in 3D space. We explore the generalization capabilities of these multi-view transformers, systematically analyzing how different approaches to camera conditioning affect model performance and robustness. Our analysis reveals that while existing attention-level conditioning methods show good generalization in terms of SE(3) transformations, they struggle when faced with varying intrinsic parameters at test time. Through careful theoretical analysis, we identify that the key to robust generalization lies in encoding the complete projective relationship between cameras, which naturally encompasses both intrinsic and extrinsic parameters while maintaining invariance to the choice of global reference frame. This insight leads to Projective Positional Encoding (PRoPE), which emerges as a natural theoretical extension of existing approaches. We demonstrate the effectiveness of PRoPE across three diverse tasks: novel view synthesis, stereo depth estimation, and multi-view spatial cognition. Our method shows strong performance and robustness, particularly when handling varying focal lengths and number of views at test time. PRoPE integrates seamlessly into existing transformer architectures, including CAT3D and UniMatch, yielding consistent improvements with minimal computational overhead. The method is flash attention-friendly and will be open-sourced.