Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language
  Models
New LLM evaluation benchmarks are important to align with the rapid
development of Large Language Models (LLMs). In this work, we present Chinese
SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality
ability of language models to answer short questions, and Chinese SimpleQA
mainly has five properties (i.e., Chinese, Diverse, High-quality, Static,
Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6
major topics with 99 diverse subtopics. Second, we conduct a comprehensive
quality control process to achieve high-quality questions and answers, where
the reference answers are static and cannot be changed over time. Third,
following SimpleQA, the questions and answers are very short, and the grading
process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we
perform a comprehensive evaluation on the factuality abilities of existing
LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to
better understand the Chinese factuality abilities of their models and
facilitate the growth of foundation models.
6732da1ff014ee495569041b
Yancheng He
6732da1ff014ee495569041c
Shilong Li
claimed_verified
2024-12-19T18:05:05.132Z
64d8cfbe99eb22b8da673150
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9gXtURPqQRzR4cyUiDkHT.jpeg
Shilong Li
user
w332323
6732da1ff014ee495569041d
Jiaheng Liu
claimed_verified
2025-01-05T23:03:18.791Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6732da1ff014ee495569041e
Yingshui Tan
6732da1ff014ee495569041f
Hui Huang
6732da1ff014ee4955690420
Weixun Wang
6732da1ff014ee4955690421
Xingyuan Bu
claimed_verified
2025-04-09T14:37:25.640Z
6444e7765691ca69b0d95856
https://cdn-avatars.huggingface.co/v1/production/uploads/6444e7765691ca69b0d95856/WbtUXVbC2VIribtVAsLp7.jpeg
Xingyuan Bu
user
sefira32
6732da1ff014ee4955690422
Hangyu Guo
6732da1ff014ee4955690423
Chengwei Hu
6732da1ff014ee4955690424
Boren Zheng
6732da1ff014ee4955690425
Xuepeng Liu
6732da1ff014ee4955690426
Dekai Sun
6732da1ff014ee4955690427
Wenbo Su
6732da1ff014ee4955690428
Bo Zheng
Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language
  Models
New LLM evaluation benchmarks are important to align with the rapid
development of Large Language Models (LLMs). In this work, we present Chinese
SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality
ability of language models to answer short questions, and Chinese SimpleQA
mainly has five properties (i.e., Chinese, Diverse, High-quality, Static,
Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6
major topics with 99 diverse subtopics. Second, we conduct a comprehensive
quality control process to achieve high-quality questions and answers, where
the reference answers are static and cannot be changed over time. Third,
following SimpleQA, the questions and answers are very short, and the grading
process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we
perform a comprehensive evaluation on the factuality abilities of existing
LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to
better understand the Chinese factuality abilities of their models and
facilitate the growth of foundation models.
Chinese SimpleQA is a comprehensive benchmark focusing on evaluating the factuality of LLMs for Chinese short questions through diverse and high-quality data.
Large Language Models LLM Chinese SimpleQA factuality ability comprehensive benchmark quality control reference answers grading process OpenAI API