HumanSense: From Multimodal Perception to Empathetic Context-Aware
  Responses through Reasoning MLLMs
While Multimodal Large Language Models (MLLMs) show immense promise for
achieving truly human-like interactions, progress is hindered by the lack of
fine-grained evaluation frameworks for human-centered scenarios, encompassing
both the understanding of complex human intentions and the provision of
empathetic, context-aware responses. Here we introduce HumanSense, a
comprehensive benchmark designed to evaluate the human-centered perception and
interaction capabilities of MLLMs, with a particular focus on deep
understanding of extended multimodal contexts and the formulation of rational
feedback. Our evaluation reveals that leading MLLMs still have considerable
room for improvement, particularly for advanced interaction-oriented tasks.
Supplementing visual input with audio and text information yields substantial
improvements, and Omni-modal models show advantages on these tasks.
Furthermore, we argue that appropriate feedback stems from a contextual
analysis of the interlocutor's needs and emotions, with reasoning ability
serving as the key to unlocking it. Accordingly, we employ a multi-stage,
modality-progressive reinforcement learning to enhance the reasoning abilities
of an Omni model, achieving substantial gains on evaluation results.
Additionally, we observe that successful reasoning processes exhibit highly
consistent thought patterns. By designing corresponding prompts, we also
enhance the performance of non-reasoning models in a training-free manner.
Project page:
brightpinkhttps://digital-avatar.github.io/ai/HumanSense/
689e9eb4a4caabb4320e5d3a
Zheng Qin
689e9eb4a4caabb4320e5d3b
Ruobing Zheng
689e9eb4a4caabb4320e5d3c
Yabing Wang
689e9eb4a4caabb4320e5d3d
Tianqi Li
689e9eb4a4caabb4320e5d3e
Yi Yuan
689e9eb4a4caabb4320e5d3f
Jingdong Chen
689e9eb4a4caabb4320e5d40
Le Wang
HumanSense: From Multimodal Perception to Empathetic Context-Aware
  Responses through Reasoning MLLMs
While Multimodal Large Language Models (MLLMs) show immense promise for
achieving truly human-like interactions, progress is hindered by the lack of
fine-grained evaluation frameworks for human-centered scenarios, encompassing
both the understanding of complex human intentions and the provision of
empathetic, context-aware responses. Here we introduce HumanSense, a
comprehensive benchmark designed to evaluate the human-centered perception and
interaction capabilities of MLLMs, with a particular focus on deep
understanding of extended multimodal contexts and the formulation of rational
feedback. Our evaluation reveals that leading MLLMs still have considerable
room for improvement, particularly for advanced interaction-oriented tasks.
Supplementing visual input with audio and text information yields substantial
improvements, and Omni-modal models show advantages on these tasks.
Furthermore, we argue that appropriate feedback stems from a contextual
analysis of the interlocutor's needs and emotions, with reasoning ability
serving as the key to unlocking it. Accordingly, we employ a multi-stage,
modality-progressive reinforcement learning to enhance the reasoning abilities
of an Omni model, achieving substantial gains on evaluation results.
Additionally, we observe that successful reasoning processes exhibit highly
consistent thought patterns. By designing corresponding prompts, we also
enhance the performance of non-reasoning models in a training-free manner.
Project page:
brightpinkhttps://digital-avatar.github.io/ai/HumanSense/
HumanSense is a benchmark for evaluating human-centered perception and interaction in Multimodal Large Language Models, focusing on multimodal context understanding and rational feedback through reinforcement learning.
Multimodal Large Language Models HumanSense benchmark human-centered perception interaction capabilities extended multimodal contexts rational feedback reinforcement learning Omni-modal models contextual analysis reasoning abilities thought patterns prompts