Task-Aware Routing of Representations for Invariant–Equivariant Self-Supervised Learning
Jaebyeong Jeon, Hyeonseo Jang, Jy-yong Sohn, Kibok Lee
Equivariant representation learning aims to capture variations induced by input transformations in the representation space, whereas invariant representation learning focuses on encoding semantic information by disregarding such transformations. Recent studies have shown that jointly learning both types of representations is often beneficial for downstream tasks, typically achieved by employing separate projection heads. However, we argue that there exists essential information shared between invariant and equivariant learning, such that using independent projection heads leads to representational redundancy and suboptimal utilization of model capacity. To address this, we propose a task-aware routing mechanism that enables the model to dynamically extract task-relevant information from multiple projection heads. This encourages the projection heads to specialize as either shared or task-specific experts, resulting in reduced representational redundancy, as evidenced by lower canonical correlations between invariant and equivariant representations. Experimental results demonstrate the effectiveness of our approach across various transfer learning tasks. The code will be released.