Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and
  Voice
Human social behaviors are inherently multimodal necessitating the
development of powerful audiovisual models for their perception. In this paper,
we present Social-MAE, our pre-trained audiovisual Masked Autoencoder based on
an extended version of Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE),
which is pre-trained on audiovisual social data. Specifically, we modify
CAV-MAE to receive a larger number of frames as input and pre-train it on a
large dataset of human social interaction (VoxCeleb2) in a self-supervised
manner. We demonstrate the effectiveness of this model by finetuning and
evaluating the model on different social and affective downstream tasks,
namely, emotion recognition, laughter detection and apparent personality
estimation. The model achieves state-of-the-art results on multimodal emotion
recognition and laughter recognition and competitive results for apparent
personality estimation, demonstrating the effectiveness of in-domain
self-supervised pre-training. Code and model weight are available here
https://github.com/HuBohy/SocialMAE.
68b1a26c851c6e7b001ec555
Hugo Bohy
claimed_verified
2025-09-01T07:53:11.974Z
63bc1c97d8d676a2299e7c60
https://cdn-avatars.huggingface.co/v1/production/uploads/63bc1c97d8d676a2299e7c60/4pNSVkG3zAItcaCmlmOiE.png
Hugo Bohy
user
HuBohy
68b1a26c851c6e7b001ec556
Minh Tran
68b1a26c851c6e7b001ec557
Kevin El Haddad
68b1a26c851c6e7b001ec558
Thierry Dutoit
68b1a26c851c6e7b001ec559
Mohammad Soleymani
Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and
  Voice
Human social behaviors are inherently multimodal necessitating the
development of powerful audiovisual models for their perception. In this paper,
we present Social-MAE, our pre-trained audiovisual Masked Autoencoder based on
an extended version of Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE),
which is pre-trained on audiovisual social data. Specifically, we modify
CAV-MAE to receive a larger number of frames as input and pre-train it on a
large dataset of human social interaction (VoxCeleb2) in a self-supervised
manner. We demonstrate the effectiveness of this model by finetuning and
evaluating the model on different social and affective downstream tasks,
namely, emotion recognition, laughter detection and apparent personality
estimation. The model achieves state-of-the-art results on multimodal emotion
recognition and laughter recognition and competitive results for apparent
personality estimation, demonstrating the effectiveness of in-domain
self-supervised pre-training. Code and model weight are available here
https://github.com/HuBohy/SocialMAE.
Social-MAE, an extended Contrastive Audio-Visual Masked Auto-Encoder, achieves state-of-the-art performance in multimodal emotion and laughter recognition through in-domain self-supervised pre-training.
Masked Autoencoder Contrastive Audio-Visual Masked Auto-Encoder self-supervised learning emotion recognition laughter detection apparent personality estimation