DUET: Dual-Facet Pseudo Labeling and Uncertainty-aware Exploration & Exploitation Training for Source-Free Adaptation
JaeYun Lee, Jae Hyeon Park, Gyoomin Lee, Bogyeong Kim, Min Hee Cha, Hyeok Nam, Joo Jeon, Hyunse Lee, Sung In Cho
Source-free domain adaptation (SFDA) aims to adapt a pre-trained source model to an unlabeled target domain without requiring labeled source data. In a self-supervised manner, relying on pseudo labels on target domain samples facilitates the domain adaptation performance providing strong supervision.However, the critical problem of this approach is the inherent instability of the pre-trained source model in the target domain, leading to unreliable pseudo labels for the target domain data. To tackle this, we propose a novel Dual-facet pseudo labeling strategy that jointly leverages a task-specific perspective and a domain-invariant perspective, assigning pseudo labels only to target samples with agreement between a target model and CLIP. To further enhance representation learning without introducing noisy supervision, we apply consistency training to uncertain samples. Additionally, we introduce a Tsallis mutual information (TMI)-based vision optimization strategy guided by an Uncertainty-based adaptation index (UAI), which dynamically modulates entropy sensitivity based on the model’s adaptation uncertainty. The UAI-based training paradigm enables stable and adaptive domain alignment by effectively balancing exploration and exploitation processes during the optimization process. Our proposed method achieves state-of-the-art performance on domain adaptation benchmark datasets, improving adaptation accuracy by 1.6\% on Office-Home, 1.4\% on VisDA-C, and 2.9\% on DomainNet-126, demonstrating its effectiveness in SFDA.