Improving Task-Specific Multimodal Sentiment Analysis with General MLLMs via Prompting
Haoyu Zhang, Yinan Zhang, Chaolong Ying, Xiaoying Tang, Tianshu Yu
Multimodal Sentiment Analysis (MSA) aims to predict sentiment from diverse data types, such as video, audio, and language. Recent progress in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance across various tasks. However, in MSA, the increase in computational costs does not always correspond to a significant improvement in performance, raising concerns about the cost-effectiveness of applying MLLMs to MSA. This paper introduces the MLLM-Guided Multimodal Sentiment Learning Framework (MMSLF). It improves the performance of task-specific MSA models by leveraging the generalized knowledge of MLLMs through a teacher-student framework, rather than directly using MLLMs for sentiment prediction. The proposed teacher, equipped with a powerful MLLM (e.g., GPT-4o-mini), guides the student model by generating context-aware prompts, facilitating better alignment and learning of multimodal sentiment features. Extensive experiments on the SIMS, MOSI, and MOSEI datasets demonstrate that our framework enables task-specific models to achieve state-of-the-art performance across most metrics. This also provides new insights into the application of general MLLMs for improving MSA.