MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes
Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, Sanglu Lu, Hongkai Wen
Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object detection, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. To capture such region-related features,  SignGraph model extracts the cross-region features by building  Local Sign Graph (LSG) module, Temporal Sign Graph (TSG) module. However, we emphasize that although capturing cross-region dependencies can improve sign language performance, it may degrade the representation quality of local regions.To mitigate this,  We introduce MixSignGraph, which represents sign sequences as a group of mixed graphs and designs the following graph modules for feature extraction. Specifically, besides from  the LSG module and TSG module that model the intra-frame and inter-frame cross-regions features, we design a simple  yet effective Hierarchical Sign Graph (HSG) module, which enhances local region representations following the extraction of cross-region features, by  aggregating the same-region features from different-granularity feature maps of a frame, \ie to boosts discriminative local features.In addition, to further improve the performance of gloss-free sign language tasks, we propose a simple yet counter-intuitive Text-based CTC Pre-training (TCTC) method, which generates pseudo gloss labels from text sequence for model pre-training. Extensive experiments conducted on current five public sign language datasets demonstrate that our model surpasses the SOTA models on multiple sign language tasks across several datasets, without relying on any additional cues. Code and models are made available after publication.