LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large
  Multimodal Models
With the rapid development of AI-generated content, the future internet may
be inundated with synthetic data, making the discrimination of authentic and
credible multimodal data increasingly challenging. Synthetic data detection has
thus garnered widespread attention, and the performance of large multimodal
models (LMMs) in this task has attracted significant interest. LMMs can provide
natural language explanations for their authenticity judgments, enhancing the
explainability of synthetic content detection. Simultaneously, the task of
distinguishing between real and synthetic data effectively tests the
perception, knowledge, and reasoning capabilities of LMMs. In response, we
introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to
detect synthetic data across multiple modalities. LOKI encompasses video,
image, 3D, text, and audio modalities, comprising 18K carefully curated
questions across 26 subcategories with clear difficulty levels. The benchmark
includes coarse-grained judgment and multiple-choice questions, as well as
fine-grained anomaly selection and explanation tasks, allowing for a
comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6
closed-source models on LOKI, highlighting their potential as synthetic data
detectors and also revealing some limitations in the development of LMM
capabilities. More information about LOKI can be found at
https://opendatalab.github.io/LOKI/
670dd110aa00e89d82f7011f
Junyan Ye
admin_assigned
2024-10-15T09:28:39.560Z
66978ee0b8656f6506b4acb2
/avatars/298acb8222e189fce4368985ee5374a1.svg
Junyan Ye
user
Yejy53
670dd110aa00e89d82f70120
Baichuan Zhou
claimed_verified
2024-10-15T09:10:58.373Z
646325085897b675c65aea0f
/avatars/28ce7388f9318b49bdd0a5594c0f6732.svg
Baichuan Zhou
user
bczhou
670dd110aa00e89d82f70121
Zilong Huang
claimed_verified
2024-10-16T09:44:20.395Z
6487e158f675b4a7867f45fa
https://cdn-avatars.huggingface.co/v1/production/uploads/6487e158f675b4a7867f45fa/J0sls6zZ682o-SH7iQs7B.jpeg
Zilong Huang
user
SereinH
670dd110aa00e89d82f70122
Junan Zhang
claimed_verified
2025-09-12T16:27:20.977Z
63f8979cb0ae1748524be22c
https://cdn-avatars.huggingface.co/v1/production/uploads/1677236096253-noauth.png
junan zhang
user
viewfinder-annn
670dd110aa00e89d82f70123
Tianyi Bai
claimed_verified
2024-10-15T09:11:00.531Z
641802c31f1f3b0fa812225c
https://cdn-avatars.huggingface.co/v1/production/uploads/641802c31f1f3b0fa812225c/i-OFWm8d37ofsmuvvAKYR.png
becca_bai
user
beccabai
670dd110aa00e89d82f70124
Hengrui Kang
admin_assigned
2024-10-15T13:24:34.153Z
6617f09e4f9cea969619bfbd
/avatars/4dbc9c7b6814edcdab334ed3f0a1a4dd.svg
Heng Rui Kang
user
ruikang2002
670dd110aa00e89d82f70125
Jun He
claimed_verified
2024-10-15T13:39:45.836Z
670ddb69d6ac6394419d88c5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XxnGNaX3FWug4aiZVjg93.png
Jun He
user
JunHe0915
670dd110aa00e89d82f70126
Honglin Lin
670dd110aa00e89d82f70127
Zihao Wang
670dd110aa00e89d82f70128
Tong Wu
claimed_verified
2024-12-11T09:21:45.971Z
6310929fda0aaad2f70397a1
https://cdn-avatars.huggingface.co/v1/production/uploads/6310929fda0aaad2f70397a1/YoHeYZezmCOXwL-qsiZOc.png
Tong Wu
user
wutong16
670dd110aa00e89d82f70129
Zhizheng Wu
670dd110aa00e89d82f7012a
Yiping Chen
admin_assigned
2024-10-15T13:22:50.336Z
653a22c6a6e8c78bc13f9d3c
/avatars/c9ca843aded54960c661f9eb232d6d60.svg
Yiping Chen
user
chenchenchenchenya
670dd110aa00e89d82f7012b
Dahua Lin
admin_assigned
2024-10-15T13:22:28.798Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
670dd110aa00e89d82f7012c
Conghui He
admin_assigned
2024-10-15T13:22:43.987Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
670dd110aa00e89d82f7012d
Weijia Li
admin_assigned
2024-10-15T13:22:37.451Z
66d5b56c77a026c3d2086a79
/avatars/45da07fd82fd455955faa05b27a6393f.svg
Weijia Li
user
liweijia
LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large
  Multimodal Models
With the rapid development of AI-generated content, the future internet may
be inundated with synthetic data, making the discrimination of authentic and
credible multimodal data increasingly challenging. Synthetic data detection has
thus garnered widespread attention, and the performance of large multimodal
models (LMMs) in this task has attracted significant interest. LMMs can provide
natural language explanations for their authenticity judgments, enhancing the
explainability of synthetic content detection. Simultaneously, the task of
distinguishing between real and synthetic data effectively tests the
perception, knowledge, and reasoning capabilities of LMMs. In response, we
introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to
detect synthetic data across multiple modalities. LOKI encompasses video,
image, 3D, text, and audio modalities, comprising 18K carefully curated
questions across 26 subcategories with clear difficulty levels. The benchmark
includes coarse-grained judgment and multiple-choice questions, as well as
fine-grained anomaly selection and explanation tasks, allowing for a
comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6
closed-source models on LOKI, highlighting their potential as synthetic data
detectors and also revealing some limitations in the development of LMM
capabilities. More information about LOKI can be found at
https://opendatalab.github.io/LOKI/
LOKI is a benchmark for evaluating the synthetic data detection capabilities of large multimodal models across various modalities.
multimodal models synthetic data detection explainability natural language explanations video image 3D text audio benchmark coarse-grained judgment multiple-choice questions fine-grained anomaly selection explanation tasks