Doodle to Detect: A Goofy but Powerful Approach to Skeleton-based Hand Gesture Recognition
Sang Hoon Han, Seonho Lee, Hyeok Nam, Jae Hyeon Park, Min Hee Cha, Min Geol Kim, Hyunse Lee, Sangyeon Ahn, Moonju Chae, Sung In Cho
Skeleton-based hand gesture recognition plays a crucial role in enabling intuitive human–computer interaction. Traditional skeleton-based gesture recognition methods have primarily relied on hand-crafted features—such as distances between joint coordinates within the same frame or positional changes of joints across different frames—to alleviate issues arising from variations in sensor viewpoints or individual body proportions. However, hand-crafted features fail to capture the comprehensive information embedded in raw skeleton data required for effective hand gesture recognition, and they exhibit poor interpretability. Moreover, the normalization process in traditional methods, which depends on training data, introduces a domain gap between training and testing data, thereby posing challenges in applying the trained model to diverse test environments. To overcome these issues, our study proposes a novel approach for real-time gesture detection in skeleton-based hand gesture recognition. Specifically, we exclude traditional handcrafted features and introduce a new method called Skeleton Kinematics Extraction Through Coordinated grapH (SKETCH), which transforms four-dimensional (time, x, y, and z) skeleton raw data into a simple and intuitive visual representation. By applying our novel Dynamic Range Embedding (DRE), which encodes axis-wise relative movement range information from each x, y, and z coordinate, SKETCH produces a graph image that richly captures the raw data’s inherent information. The pre-trained vision model is then fine-tuned using the SKETCH to detect and classify hand gestures. The visual graph facilitates direct interpretation of the model’s focus within the raw input data. Moreover, the proposed method performs independent min-max normalization on a fixed-length segment of the incoming sequence in real time, mitigating performance degradation due to fluctuations in absolute coordinates caused by differences in sensor viewpoints and individual body proportions. Experimental results demonstrate that our approach outperforms existing methods on the SHREC’19 and SHREC’22 benchmarks while ensuring both robustness and interpretability.