SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy
  Segment Optimization
Social relation reasoning aims to identify relation categories such as
friends, spouses, and colleagues from images. While current methods adopt the
paradigm of training a dedicated network end-to-end using labeled image data,
they are limited in terms of generalizability and interpretability. To address
these issues, we first present a simple yet well-crafted framework named
{\name}, which combines the perception capability of Vision Foundation Models
(VFMs) and the reasoning capability of Large Language Models (LLMs) within a
modular framework, providing a strong baseline for social relation recognition.
Specifically, we instruct VFMs to translate image content into a textual social
story, and then utilize LLMs for text-based reasoning. {\name} introduces
systematic design principles to adapt VFMs and LLMs separately and bridge their
gaps. Without additional model training, it achieves competitive zero-shot
results on two databases while offering interpretable answers, as LLMs can
generate language-based explanations for the decisions. The manual prompt
design process for LLMs at the reasoning phase is tedious and an automated
prompt optimization method is desired. As we essentially convert a visual
classification task into a generative task of LLMs, automatic prompt
optimization encounters a unique long prompt optimization issue. To address
this issue, we further propose the Greedy Segment Prompt Optimization (GSPO),
which performs a greedy search by utilizing gradient information at the segment
level. Experimental results show that GSPO significantly improves performance,
and our method also generalizes to different image styles. The code is
available at https://github.com/Mengzibin/SocialGPT.
6721931db02ab0e490b45411
Wanhua Li
claimed_verified
2024-10-30T09:52:46.803Z
658bb7e47459b6e471b9d2e6
/avatars/efd8051b468b4dbcb5d149479de67c58.svg
Wanhua Li
user
EthanTaylor
6721931db02ab0e490b45412
Zibin Meng
claimed_verified
2024-10-30T09:52:48.983Z
64d3a3a29d84115700581720
/avatars/db23c6585975e08c6629393538ef8980.svg
Zibin Meng
user
zmeng0116
6721931db02ab0e490b45413
Jiawei Zhou
6721931db02ab0e490b45414
Donglai Wei
6721931db02ab0e490b45415
Chuang Gan
6721931db02ab0e490b45416
Hanspeter Pfister
SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy
  Segment Optimization
Social relation reasoning aims to identify relation categories such as
friends, spouses, and colleagues from images. While current methods adopt the
paradigm of training a dedicated network end-to-end using labeled image data,
they are limited in terms of generalizability and interpretability. To address
these issues, we first present a simple yet well-crafted framework named
{\name}, which combines the perception capability of Vision Foundation Models
(VFMs) and the reasoning capability of Large Language Models (LLMs) within a
modular framework, providing a strong baseline for social relation recognition.
Specifically, we instruct VFMs to translate image content into a textual social
story, and then utilize LLMs for text-based reasoning. {\name} introduces
systematic design principles to adapt VFMs and LLMs separately and bridge their
gaps. Without additional model training, it achieves competitive zero-shot
results on two databases while offering interpretable answers, as LLMs can
generate language-based explanations for the decisions. The manual prompt
design process for LLMs at the reasoning phase is tedious and an automated
prompt optimization method is desired. As we essentially convert a visual
classification task into a generative task of LLMs, automatic prompt
optimization encounters a unique long prompt optimization issue. To address
this issue, we further propose the Greedy Segment Prompt Optimization (GSPO),
which performs a greedy search by utilizing gradient information at the segment
level. Experimental results show that GSPO significantly improves performance,
and our method also generalizes to different image styles. The code is
available at https://github.com/Mengzibin/SocialGPT.
A modular framework that leverages Vision Foundation Models and Large Language Models for social relation reasoning achieves competitive zero-shot accuracy and interpretability through text-based explanations, with automatic prompt optimization to enhance performance.
Vision Foundation Models Large Language Models textual social story text-based reasoning systematic design principles zero-shot results automatic prompt optimization Greedy Segment Prompt Optimization gradient information segment level long prompt optimization