MLLM-as-a-Judge for Image Safety without Human Labeling
Image content safety has become a significant challenge with the rise of
visual media on online platforms. Meanwhile, in the age of AI-generated content
(AIGC), many image generation models are capable of producing harmful content,
such as images containing sexual or violent material. Thus, it becomes crucial
to identify such unsafe images based on established safety rules. Pre-trained
Multimodal Large Language Models (MLLMs) offer potential in this regard, given
their strong pattern recognition abilities. Existing approaches typically
fine-tune MLLMs with human-labeled datasets, which however brings a series of
drawbacks. First, relying on human annotators to label data following intricate
and detailed guidelines is both expensive and labor-intensive. Furthermore,
users of safety judgment systems may need to frequently update safety rules,
making fine-tuning on human-based annotation more challenging. This raises the
research question: Can we detect unsafe images by querying MLLMs in a zero-shot
setting using a predefined safety constitution (a set of safety rules)? Our
research showed that simply querying pre-trained MLLMs does not yield
satisfactory results. This lack of effectiveness stems from factors such as the
subjectivity of safety rules, the complexity of lengthy constitutions, and the
inherent biases in the models. To address these challenges, we propose a
MLLM-based method includes objectifying safety rules, assessing the relevance
between rules and images, making quick judgments based on debiased token
probabilities with logically complete yet simplified precondition chains for
safety rules, and conducting more in-depth reasoning with cascaded
chain-of-thought processes if necessary. Experiment results demonstrate that
our method is highly effective for zero-shot image safety judgment tasks.
6777591138f9a731d4f575bb
Zhenting Wang
claimed_verified
2025-01-03T14:03:15.648Z
64dfcc62e8b6f3f3baa950e0
/avatars/21bbff67d46c08044efe2406575aa77e.svg
Zhenting Wang
user
ztwang
6777591138f9a731d4f575bc
Shuming Hu
admin_assigned
2025-01-06T14:53:03.995Z
6563c6846ef2a1d0f989f11b
/avatars/b03f6680ac3e718a1c38fec59a211b62.svg
Shuming Hu
user
shumingh
6777591138f9a731d4f575bd
Shiyu Zhao
admin_assigned
2025-01-06T14:53:11.798Z
667caf5baeba4a9f63860cf3
https://cdn-avatars.huggingface.co/v1/production/uploads/667caf5baeba4a9f63860cf3/p4D4IXN1JgHwI8RGe5p6F.jpeg
Shiyu Zhao
user
xiaofeng-94
6777591138f9a731d4f575be
Xiaowen Lin
6777591138f9a731d4f575bf
Felix Juefei-Xu
6777591138f9a731d4f575c0
Zhuowei Li
6777591138f9a731d4f575c1
Ligong Han
admin_assigned
2025-01-06T14:54:08.096Z
631e56ecf318ed8dfd3738cc
/avatars/df4f71ac795cb8c251db7f30529dc2f8.svg
Ligong Han
user
ligongh
6777591138f9a731d4f575c2
Harihar Subramanyam
admin_assigned
2025-01-06T14:54:14.540Z
6303da377373aacccd868d0f
/avatars/88813105f863a0b2125fbe8bfffdec0e.svg
Harihar Subramanyam
user
Harihar
6777591138f9a731d4f575c3
Li Chen
6777591138f9a731d4f575c4
Jianfa Chen
6777591138f9a731d4f575c5
Nan Jiang
6777591138f9a731d4f575c6
Lingjuan Lyu
6777591138f9a731d4f575c7
Shiqing Ma
admin_assigned
2025-01-06T14:56:30.168Z
644f91f317b6189cda55deea
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/gX6s1BFk_JRKq4_HSQIcU.png
Shiqing Ma
user
Kuingsmile
6777591138f9a731d4f575c8
Dimitris N. Metaxas
6777591138f9a731d4f575c9
Ankit Jain
MLLM-as-a-Judge for Image Safety without Human Labeling
Image content safety has become a significant challenge with the rise of
visual media on online platforms. Meanwhile, in the age of AI-generated content
(AIGC), many image generation models are capable of producing harmful content,
such as images containing sexual or violent material. Thus, it becomes crucial
to identify such unsafe images based on established safety rules. Pre-trained
Multimodal Large Language Models (MLLMs) offer potential in this regard, given
their strong pattern recognition abilities. Existing approaches typically
fine-tune MLLMs with human-labeled datasets, which however brings a series of
drawbacks. First, relying on human annotators to label data following intricate
and detailed guidelines is both expensive and labor-intensive. Furthermore,
users of safety judgment systems may need to frequently update safety rules,
making fine-tuning on human-based annotation more challenging. This raises the
research question: Can we detect unsafe images by querying MLLMs in a zero-shot
setting using a predefined safety constitution (a set of safety rules)? Our
research showed that simply querying pre-trained MLLMs does not yield
satisfactory results. This lack of effectiveness stems from factors such as the
subjectivity of safety rules, the complexity of lengthy constitutions, and the
inherent biases in the models. To address these challenges, we propose a
MLLM-based method includes objectifying safety rules, assessing the relevance
between rules and images, making quick judgments based on debiased token
probabilities with logically complete yet simplified precondition chains for
safety rules, and conducting more in-depth reasoning with cascaded
chain-of-thought processes if necessary. Experiment results demonstrate that
our method is highly effective for zero-shot image safety judgment tasks.
A method using Multimodal Large Language Models for zero-shot detection of unsafe images improves upon human-labeled fine-tuning by objectifying safety rules and using cascaded reasoning for complex judgments.
Multimodal Large Language Models zero-shot setting safety rules token probabilities precondition chains cascaded chain-of-thought processes