Rotary Masked Autoencoders are Versatile Learners
Uros Zivanovic, Serafina Di Gioia, Andre Scaffidi, Martín de los Rios, Gabriella Contardo, Roberto Trotta
Applying Transformers to irregular time-series typically requires specializations to their baseline architecture, which  can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), utilizing the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is an extension to the Masked Autoencoder (MAE) that enables representation learning with continuous positional information while avoiding any time-series-specific architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multi-variate time-series, images, and audio, showing that RoMAE surpasses specialized time-series architectures on difficult datasets such as the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. We additionally investigate RoMAE's ability to reconstruct the embedded continuous positions. We also demonstrate that including learned embeddings in the input sequence breaks RoPE's relative position property.