Revisiting Modeling and Evaluation Approaches in Speech Emotion
  Recognition: Considering Subjectivity of Annotators and Ambiguity of Emotions
Over the past two decades, speech emotion recognition (SER) has received
growing attention. To train SER systems, researchers collect emotional speech
databases annotated by crowdsourced or in-house raters who select emotions from
predefined categories. However, disagreements among raters are common.
Conventional methods treat these disagreements as noise, aggregating labels
into a single consensus target. While this simplifies SER as a single-label
task, it ignores the inherent subjectivity of human emotion perception. This
dissertation challenges such assumptions and asks: (1) Should minority
emotional ratings be discarded? (2) Should SER systems learn from only a few
individuals' perceptions? (3) Should SER systems predict only one emotion per
sample?
  Psychological studies show that emotion perception is subjective and
ambiguous, with overlapping emotional boundaries. We propose new modeling and
evaluation perspectives: (1) Retain all emotional ratings and represent them
with soft-label distributions. Models trained on individual annotator ratings
and jointly optimized with standard SER systems improve performance on
consensus-labeled tests. (2) Redefine SER evaluation by including all emotional
data and allowing co-occurring emotions (e.g., sad and angry). We propose an
``all-inclusive rule'' that aggregates all ratings to maximize diversity in
label representation. Experiments on four English emotion databases show
superior performance over majority and plurality labeling. (3) Construct a
penalization matrix to discourage unlikely emotion combinations during
training. Integrating it into loss functions further improves performance.
Overall, embracing minority ratings, multiple annotators, and multi-emotion
predictions yields more robust and human-aligned SER systems.
68e61d31975ac4c405ef21c9
Huang-Cheng Chou
claimed_verified
2025-10-08T11:49:00.843Z
67c912c26ec61b19b174dde3
https://cdn-avatars.huggingface.co/v1/production/uploads/67c912c26ec61b19b174dde3/C6NYw3PivMvysKm8GL4_L.jpeg
Huang-Cheng Chou
user
huangchengchou
68e61d31975ac4c405ef21ca
Chi-Chun Lee
Revisiting Modeling and Evaluation Approaches in Speech Emotion
  Recognition: Considering Subjectivity of Annotators and Ambiguity of Emotions
Over the past two decades, speech emotion recognition (SER) has received
growing attention. To train SER systems, researchers collect emotional speech
databases annotated by crowdsourced or in-house raters who select emotions from
predefined categories. However, disagreements among raters are common.
Conventional methods treat these disagreements as noise, aggregating labels
into a single consensus target. While this simplifies SER as a single-label
task, it ignores the inherent subjectivity of human emotion perception. This
dissertation challenges such assumptions and asks: (1) Should minority
emotional ratings be discarded? (2) Should SER systems learn from only a few
individuals' perceptions? (3) Should SER systems predict only one emotion per
sample?
  Psychological studies show that emotion perception is subjective and
ambiguous, with overlapping emotional boundaries. We propose new modeling and
evaluation perspectives: (1) Retain all emotional ratings and represent them
with soft-label distributions. Models trained on individual annotator ratings
and jointly optimized with standard SER systems improve performance on
consensus-labeled tests. (2) Redefine SER evaluation by including all emotional
data and allowing co-occurring emotions (e.g., sad and angry). We propose an
``all-inclusive rule'' that aggregates all ratings to maximize diversity in
label representation. Experiments on four English emotion databases show
superior performance over majority and plurality labeling. (3) Construct a
penalization matrix to discourage unlikely emotion combinations during
training. Integrating it into loss functions further improves performance.
Overall, embracing minority ratings, multiple annotators, and multi-emotion
predictions yields more robust and human-aligned SER systems.
Embracing minority ratings, multiple annotators, and multi-emotion predictions in speech emotion recognition improves system robustness and alignment with human perception.
speech emotion recognition emotional speech databases soft-label distributions co-occurring emotions all-inclusive rule penalization matrix loss functions