ReviewScore: Misinformed Peer Review Detection with Large Language
  Models
Peer review serves as a backbone of academic research, but in most AI
conferences, the review quality is degrading as the number of submissions
explodes. To reliably detect low-quality reviews, we define misinformed review
points as either "weaknesses" in a review that contain incorrect premises, or
"questions" in a review that can be already answered by the paper. We verify
that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce
ReviewScore indicating if a review point is misinformed. To evaluate the
factuality of each premise of weaknesses, we propose an automated engine that
reconstructs every explicit and implicit premise from a weakness. We build a
human expert-annotated ReviewScore dataset to check the ability of LLMs to
automate ReviewScore evaluation. Then, we measure human-model agreements on
ReviewScore using eight current state-of-the-art LLMs and verify moderate
agreements. We also prove that evaluating premise-level factuality shows
significantly higher agreements than evaluating weakness-level factuality. A
thorough disagreement analysis further supports a potential of fully automated
ReviewScore evaluation.
68d9f9bd0177a6054b013b5f
Hyun Ryu
claimed_verified
2025-09-29T05:57:59.016Z
666d506fc0f3d5afc24dd5ca
/avatars/eeb98947415d08a26815fd139c76a071.svg
Hyun Ryu
user
hyun1905
68d9f9bd0177a6054b013b60
Doohyuk Jang
claimed_verified
2025-09-29T11:26:32.034Z
62845957b410bd779033759c
/avatars/4feef73c06f2f7de6abf7a4789ac13f9.svg
Doohyuk Jang
user
jadohu
68d9f9bd0177a6054b013b61
Hyemin S. Lee
68d9f9bd0177a6054b013b62
Joonhyun Jeong
68d9f9bd0177a6054b013b63
Gyeongman Kim
68d9f9bd0177a6054b013b64
Donghyeon Cho
68d9f9bd0177a6054b013b65
Gyouk Chu
68d9f9bd0177a6054b013b66
Minyeong Hwang
68d9f9bd0177a6054b013b67
Hyeongwon Jang
68d9f9bd0177a6054b013b68
Changhun Kim
68d9f9bd0177a6054b013b69
Haechan Kim
68d9f9bd0177a6054b013b6a
Jina Kim
68d9f9bd0177a6054b013b6b
Joowon Kim
claimed_verified
2025-09-29T05:57:56.550Z
6666b61eaf95872a03a0a673
/avatars/fc0c144cf6307357d45d7ca2d6ba8d2f.svg
Joowon
user
kjwispro
68d9f9bd0177a6054b013b6c
Yoonjeon Kim
68d9f9bd0177a6054b013b6d
Kwanhyung Lee
68d9f9bd0177a6054b013b6e
Chanjae Park
68d9f9bd0177a6054b013b6f
Heecheol Yun
68d9f9bd0177a6054b013b70
Gregor Betz
68d9f9bd0177a6054b013b71
Eunho Yang
ReviewScore: Misinformed Peer Review Detection with Large Language
  Models
Peer review serves as a backbone of academic research, but in most AI
conferences, the review quality is degrading as the number of submissions
explodes. To reliably detect low-quality reviews, we define misinformed review
points as either "weaknesses" in a review that contain incorrect premises, or
"questions" in a review that can be already answered by the paper. We verify
that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce
ReviewScore indicating if a review point is misinformed. To evaluate the
factuality of each premise of weaknesses, we propose an automated engine that
reconstructs every explicit and implicit premise from a weakness. We build a
human expert-annotated ReviewScore dataset to check the ability of LLMs to
automate ReviewScore evaluation. Then, we measure human-model agreements on
ReviewScore using eight current state-of-the-art LLMs and verify moderate
agreements. We also prove that evaluating premise-level factuality shows
significantly higher agreements than evaluating weakness-level factuality. A
thorough disagreement analysis further supports a potential of fully automated
ReviewScore evaluation.
An automated engine evaluates the factuality of review points in AI conference papers, demonstrating moderate agreement with human experts and higher accuracy at the premise level.
misinformed review points weaknesses questions ReviewScore automated engine premise-level factuality weakness-level factuality human expert-annotated dataset state-of-the-art LLMs human-model agreements