MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Knowledge editing techniques have emerged as essential tools for updating the
factual knowledge of large language models (LLMs) and multimodal models (LMMs),
allowing them to correct outdated or inaccurate information without retraining
from scratch. However, existing benchmarks for multimodal knowledge editing
primarily focus on entity-level knowledge represented as simple triplets, which
fail to capture the complexity of real-world multimodal information. To address
this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge
Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse
visual knowledge in real-world scenarios. MMKE-Bench addresses these
limitations by incorporating three types of editing tasks: visual entity
editing, visual semantic editing, and user-specific editing. Besides,
MMKE-Bench uses free-form natural language to represent and edit knowledge,
offering a more flexible and effective format. The benchmark consists of 2,940
pieces of knowledge and 8,363 images across 33 broad categories, with
evaluation questions automatically generated and human-verified. We assess five
state-of-the-art knowledge editing methods on three prominent LMMs, revealing
that no method excels across all criteria, and that visual and user-specific
edits are particularly challenging. MMKE-Bench sets a new standard for
evaluating the robustness of multimodal knowledge editing techniques, driving
progress in this rapidly evolving field.
67c11908dfcbe8a49cf19952
Yuntao Du
67c11908dfcbe8a49cf19953
Kailin Jiang
claimed_verified
2025-03-06T09:27:44.030Z
65745569839aa08899ea5d27
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg
kailinjiang
user
kailinjiang
67c11908dfcbe8a49cf19954
Zhi Gao
67c11908dfcbe8a49cf19955
Chenrui Shi
67c11908dfcbe8a49cf19956
Zilong Zheng
67c11908dfcbe8a49cf19957
Siyuan Qi
67c11908dfcbe8a49cf19958
Qing Li
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Knowledge editing techniques have emerged as essential tools for updating the
factual knowledge of large language models (LLMs) and multimodal models (LMMs),
allowing them to correct outdated or inaccurate information without retraining
from scratch. However, existing benchmarks for multimodal knowledge editing
primarily focus on entity-level knowledge represented as simple triplets, which
fail to capture the complexity of real-world multimodal information. To address
this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge
Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse
visual knowledge in real-world scenarios. MMKE-Bench addresses these
limitations by incorporating three types of editing tasks: visual entity
editing, visual semantic editing, and user-specific editing. Besides,
MMKE-Bench uses free-form natural language to represent and edit knowledge,
offering a more flexible and effective format. The benchmark consists of 2,940
pieces of knowledge and 8,363 images across 33 broad categories, with
evaluation questions automatically generated and human-verified. We assess five
state-of-the-art knowledge editing methods on three prominent LMMs, revealing
that no method excels across all criteria, and that visual and user-specific
edits are particularly challenging. MMKE-Bench sets a new standard for
evaluating the robustness of multimodal knowledge editing techniques, driving
progress in this rapidly evolving field.
A new benchmark evaluates the multimodal knowledge editing capabilities of large models across diverse visual tasks and formats, highlighting challenges in visual and user-specific edits.
knowledge editing techniques large language models multimodal models MMKE-Bench visual entity editing visual semantic editing user-specific editing free-form natural language evaluation criteria robustness