PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for
  Multimodal Large Language Models
In recent years, Multimodal Large Language Models (MLLMs) have demonstrated
remarkable advancements in tasks such as visual question answering, visual
understanding, and reasoning. However, this impressive progress relies on vast
amounts of data collected from the internet, raising significant concerns about
privacy and security. To address these issues, machine unlearning (MU) has
emerged as a promising solution, enabling the removal of specific knowledge
from an already trained model without requiring retraining from scratch.
Although MU for MLLMs has gained attention, current evaluations of its efficacy
remain incomplete, and the underlying problem is often poorly defined, which
hinders the development of strategies for creating more secure and trustworthy
systems. To bridge this gap, we introduce a benchmark, named PEBench, which
includes a dataset of personal entities and corresponding general event scenes,
designed to comprehensively assess the performance of MU for MLLMs. Through
PEBench, we aim to provide a standardized and robust framework to advance
research in secure and privacy-preserving multimodal models. We benchmarked 6
MU methods, revealing their strengths and limitations, and shedding light on
key challenges and opportunities for MU in MLLMs.
67d943d272843a36b74ab41c
Zhaopan Xu
admin_assigned
2025-03-19T13:50:26.175Z
646dee28174cc96d50951991
/avatars/17d88a24c905e9819268b27037015a35.svg
xu
user
xuzhaopan
67d943d272843a36b74ab41d
Pengfei Zhou
67d943d272843a36b74ab41e
Weidong Tang
admin_assigned
2025-03-19T13:50:48.159Z
67933ffb4fa1841697c9a77d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zYVbcrtZvcIC0BKXLQUBE.png
weidong.tang
user
weidongTang
67d943d272843a36b74ab41f
Jiaxin Ai
67d943d272843a36b74ab420
Wangbo Zhao
admin_assigned
2025-03-19T13:50:50.721Z
65d8b116f5d21ab1939efe03
/avatars/54d84f74faa2cc6fb8d6b58573dfde6d.svg
Wangbo Zhao
user
heisejiasuo
67d943d272843a36b74ab421
Xiaojiang Peng
admin_assigned
2025-03-19T13:50:56.476Z
64acc5b3f5d40283c620d625
/avatars/1e7f488db184f445f7d93a37f6470590.svg
Xiaojiang Peng
user
xpeng-sztu
67d943d272843a36b74ab422
Kai Wang
67d943d272843a36b74ab423
Yang You
67d943d272843a36b74ab424
Wenqi Shao
admin_assigned
2025-03-19T13:51:03.971Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
67d943d272843a36b74ab425
Hongxun Yao
67d943d272843a36b74ab426
Kaipeng Zhang
admin_assigned
2025-03-19T13:51:29.357Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for
  Multimodal Large Language Models
In recent years, Multimodal Large Language Models (MLLMs) have demonstrated
remarkable advancements in tasks such as visual question answering, visual
understanding, and reasoning. However, this impressive progress relies on vast
amounts of data collected from the internet, raising significant concerns about
privacy and security. To address these issues, machine unlearning (MU) has
emerged as a promising solution, enabling the removal of specific knowledge
from an already trained model without requiring retraining from scratch.
Although MU for MLLMs has gained attention, current evaluations of its efficacy
remain incomplete, and the underlying problem is often poorly defined, which
hinders the development of strategies for creating more secure and trustworthy
systems. To bridge this gap, we introduce a benchmark, named PEBench, which
includes a dataset of personal entities and corresponding general event scenes,
designed to comprehensively assess the performance of MU for MLLMs. Through
PEBench, we aim to provide a standardized and robust framework to advance
research in secure and privacy-preserving multimodal models. We benchmarked 6
MU methods, revealing their strengths and limitations, and shedding light on
key challenges and opportunities for MU in MLLMs.
PEBench evaluates machine unlearning methods in Multimodal Large Language Models to enhance security and privacy by addressing challenges in removing specific knowledge without full retraining.
Multimodal Large Language Models machine unlearning privacy security dataset benchmark visual question answering visual understanding reasoning personal entities general event scenes