R1-Omni: Explainable Omni-Multimodal Emotion Recognition with
  Reinforcing Learning
In this work, we present the first application of Reinforcement Learning with
Verifiable Reward (RLVR) to an Omni-multimodal large language model in the
context of emotion recognition, a task where both visual and audio modalities
play crucial roles. We leverage RLVR to optimize the Omni model, significantly
enhancing its performance in three key aspects: reasoning capability, emotion
recognition accuracy, and generalization ability. The introduction of RLVR not
only improves the model's overall performance on in-distribution data but also
demonstrates superior robustness when evaluated on out-of-distribution
datasets. More importantly, the improved reasoning capability enables clear
analysis of the contributions of different modalities, particularly visual and
audio information, in the emotion recognition process. This provides valuable
insights into the optimization of multimodal large language models.
67ce5f2389663abdbc364495
Jiaxing Zhao
admin_assigned
2025-03-10T10:12:14.110Z
6686044047f2a33570e59e31
/avatars/2656bf2cecd6d7cbffd0a912a54d25de.svg
Jiaxing Zhao
user
StarJiaxing
67ce5f2389663abdbc364496
Xihan Wei
admin_assigned
2025-03-12T10:02:49.458Z
67cfb923dfdb000d159407da
/avatars/6cc1da8f83ab7519cf240f7cb02b1b1c.svg
wei
user
xihanwei
67ce5f2389663abdbc364497
Liefeng Bo
admin_assigned
2025-03-10T10:12:31.341Z
63d0cc736b985b0f25d0412c
/avatars/3eb8c79f9a7c4c819038ea7b04e323dd.svg
Bo
user
Liefeng
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with
  Reinforcing Learning
In this work, we present the first application of Reinforcement Learning with
Verifiable Reward (RLVR) to an Omni-multimodal large language model in the
context of emotion recognition, a task where both visual and audio modalities
play crucial roles. We leverage RLVR to optimize the Omni model, significantly
enhancing its performance in three key aspects: reasoning capability, emotion
recognition accuracy, and generalization ability. The introduction of RLVR not
only improves the model's overall performance on in-distribution data but also
demonstrates superior robustness when evaluated on out-of-distribution
datasets. More importantly, the improved reasoning capability enables clear
analysis of the contributions of different modalities, particularly visual and
audio information, in the emotion recognition process. This provides valuable
insights into the optimization of multimodal large language models.
Reinforcement Learning with Verifiable Reward applied to a multimodal large language model enhances emotion recognition, reasoning, and generalization.
Reinforcement Learning with Verifiable Reward RLVR Omni-multimodal large language model emotion recognition reasoning capability generalization ability robustness out-of-distribution multimodal