UGCFormer: A Simple yet Universal Graph Cross-attention Network
Jiaming Zhuo, Ziyi Ma, Yintong Lu, Yuwei Liu, Kun Fu, Di Jin, Chuan Wang, Wu Wenning, Zhen Wang, Xiaochun Cao, Liang Yang
Graph Transformers (GTs), effective at capturing long-range dependencies and structural biases simultaneously, are gradually recognized as viable alternatives to traditional Graph Neural Networks (GNNs). Advanced methods for GTs to leverage topology information involve integrating the GNN modules and modulating node attributes using positional encodings. However, the potential mechanism for their effectiveness remains not fully explored and understood. Upon revisiting these two methods, a common mechanism—Cross Aggregation—that effectively captures the interrelation between graph topology and node attributes is revealed. Building on this insight, this paper proposes a universal GT architecture with linear complexity, named Universal Graph Cross-attention Transformer (UGCFormer). The idea is to interactively learn the representations of graph topology and node attributes via a linearized Dual Cross-attention (DCA) module. In theory, this module can adaptively capture interrelations between these two types of graph information, thus achieving effective aggregation. To mitigate the overfitting problem caused by the two-channel design, a consistency constraint is utilized to enhance their commonality, which prevents representation distortion. Extensive evaluations on benchmark datasets demonstrate the effectiveness and efficiency of UGCFormer.