Friends-MMC: A Dataset for Multi-modal Multi-party Conversation
  Understanding
Multi-modal multi-party conversation (MMC) is a less studied yet important
topic of research due to that it well fits real-world scenarios and thus
potentially has more widely-used applications. Compared with the traditional
multi-modal conversations, MMC requires stronger character-centered
understanding abilities as there are many interlocutors appearing in both the
visual and textual context. To facilitate the study of this problem, we present
Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique
utterances paired with video context. To explore the character-centered
understanding of the dialogue, we also annotate the speaker of each utterance,
the names and bounding bboxes of faces that appear in the video. Based on this
Friends-MMC dataset, we further study two fundamental MMC tasks: conversation
speaker identification and conversation response prediction, both of which have
the multi-party nature with the video or image as visual context. For
conversation speaker identification, we demonstrate the inefficiencies of
existing methods such as pre-trained models, and propose a simple yet effective
baseline method that leverages an optimization solver to utilize the context of
two modalities to achieve better performance. For conversation response
prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze
the benefits of speaker information. The code and dataset is publicly available
at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more
attention on modeling speaker information when understanding conversations.
676a3878eabbef01bb66f7f8
Yueqian Wang
676a3878eabbef01bb66f7f9
Xiaojun Meng
claimed_verified
2025-05-26T08:44:12.532Z
676bd26f295f85d93ec5dc16
/avatars/b8c00e5b9730c735b9ff3af12ed921be.svg
Meng Jason
user
mengxj08
676a3878eabbef01bb66f7fa
Yuxuan Wang
extracted_confirmed
2025-01-14T09:25:44.845Z
60b9e6837946aff342f734ae
/avatars/a711a6aa35757dfd7b78b26098a964fc.svg
Yuxuan Wang
user
ColorfulAI
676a3878eabbef01bb66f7fb
Jianxin Liang
676a3878eabbef01bb66f7fc
Qun Liu
676a3878eabbef01bb66f7fd
Dongyan Zhao
Friends-MMC: A Dataset for Multi-modal Multi-party Conversation
  Understanding
Multi-modal multi-party conversation (MMC) is a less studied yet important
topic of research due to that it well fits real-world scenarios and thus
potentially has more widely-used applications. Compared with the traditional
multi-modal conversations, MMC requires stronger character-centered
understanding abilities as there are many interlocutors appearing in both the
visual and textual context. To facilitate the study of this problem, we present
Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique
utterances paired with video context. To explore the character-centered
understanding of the dialogue, we also annotate the speaker of each utterance,
the names and bounding bboxes of faces that appear in the video. Based on this
Friends-MMC dataset, we further study two fundamental MMC tasks: conversation
speaker identification and conversation response prediction, both of which have
the multi-party nature with the video or image as visual context. For
conversation speaker identification, we demonstrate the inefficiencies of
existing methods such as pre-trained models, and propose a simple yet effective
baseline method that leverages an optimization solver to utilize the context of
two modalities to achieve better performance. For conversation response
prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze
the benefits of speaker information. The code and dataset is publicly available
at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more
attention on modeling speaker information when understanding conversations.
A Friends-MMC dataset facilitates research on multi-modal multi-party conversations by enabling studies on speaker identification and response prediction with video context.
multi-modal multi-party conversation Friends-MMC speaker identification conversation response prediction optimization solver generative dialogue models bounding bboxes character-centered understanding