Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing
Qingzhu Zhang, Jiani Zhong, Zongsheng Li, Xinke Shen, Quanying Liu
Task-specific pre-training is essential for tasks with limited and heterogeneous data. Existing task-general pre-training EEG models struggle with complex tasks like emotion recognition due to mismatches between task-specific features and broad pre-training approaches. This work aims to develop a task-specific multi-dataset joint pre-training framework for cross-dataset emotion recognition, tackling problems of large inter-dataset distribution shifts, inconsistent emotion category definitions, and substantial inter-subject variability. We introduce a Cross-Dataset covariance Alignment (CDA) loss to align second-order statistical properties across datasets, enabling robust generalization without the need for extensive labels or per-subject calibration. Additionally, we propose a hybrid encoder to capture the long-term dependency and complex dynamics of EEG, combining a Mamba-like linear attention channel encoder and a spatiotemporal dynamics model. Our method outperforms state-of-the-art large-scale EEG models by an average of 6.59% on F1 score in cross-dataset emotion recognition. Performance scales with the increase of datasets used in pre-training. Multi-dataset joint pre-training achieves a performance gain of 15.14% over single-dataset training. This work provides a scalable framework for task-specific pre-training and highlights its benefit in generalizable affective computing. Our code is available at https://anonymous.4open.science/status/Covariance_EEG_Emotion-D1C1.