ReviewerToo: Should AI Join The Program Committee? A Look At The Future
  of Peer Review
Peer review is the cornerstone of scientific publishing, yet it suffers from
inconsistencies, reviewer subjectivity, and scalability challenges. We
introduce ReviewerToo, a modular framework for studying and deploying
AI-assisted peer review to complement human judgment with systematic and
consistent assessments. ReviewerToo supports systematic experiments with
specialized reviewer personas and structured evaluation criteria, and can be
partially or fully integrated into real conference workflows. We validate
ReviewerToo on a carefully curated dataset of 1,963 paper submissions from ICLR
2025, where our experiments with the gpt-oss-120b model achieves 81.8% accuracy
for the task of categorizing a paper as accept/reject compared to 83.9% for the
average human reviewer. Additionally, ReviewerToo-generated reviews are rated
as higher quality than the human average by an LLM judge, though still trailing
the strongest expert contributions. Our analysis highlights domains where AI
reviewers excel (e.g., fact-checking, literature coverage) and where they
struggle (e.g., assessing methodological novelty and theoretical
contributions), underscoring the continued need for human expertise. Based on
these findings, we propose guidelines for integrating AI into peer-review
pipelines, showing how AI can enhance consistency, coverage, and fairness while
leaving complex evaluative judgments to domain experts. Our work provides a
foundation for systematic, hybrid peer-review systems that scale with the
growth of scientific publishing.
68ec5814cd07fb414898c8db
Gaurav Sahu
68ec5814cd07fb414898c8dc
Hugo Larochelle
68ec5814cd07fb414898c8dd
Laurent Charlin
68ec5814cd07fb414898c8de
Christopher Pal
ReviewerToo: Should AI Join The Program Committee? A Look At The Future
  of Peer Review
Peer review is the cornerstone of scientific publishing, yet it suffers from
inconsistencies, reviewer subjectivity, and scalability challenges. We
introduce ReviewerToo, a modular framework for studying and deploying
AI-assisted peer review to complement human judgment with systematic and
consistent assessments. ReviewerToo supports systematic experiments with
specialized reviewer personas and structured evaluation criteria, and can be
partially or fully integrated into real conference workflows. We validate
ReviewerToo on a carefully curated dataset of 1,963 paper submissions from ICLR
2025, where our experiments with the gpt-oss-120b model achieves 81.8% accuracy
for the task of categorizing a paper as accept/reject compared to 83.9% for the
average human reviewer. Additionally, ReviewerToo-generated reviews are rated
as higher quality than the human average by an LLM judge, though still trailing
the strongest expert contributions. Our analysis highlights domains where AI
reviewers excel (e.g., fact-checking, literature coverage) and where they
struggle (e.g., assessing methodological novelty and theoretical
contributions), underscoring the continued need for human expertise. Based on
these findings, we propose guidelines for integrating AI into peer-review
pipelines, showing how AI can enhance consistency, coverage, and fairness while
leaving complex evaluative judgments to domain experts. Our work provides a
foundation for systematic, hybrid peer-review systems that scale with the
growth of scientific publishing.
ReviewerToo, a modular AI-assisted peer review framework, complements human judgment with systematic assessments, achieving high accuracy and quality in specific domains while highlighting areas where human expertise remains essential.
AI-assisted peer review reviewer personas structured evaluation criteria gpt-oss-120b accuracy LLM judge methodological novelty theoretical contributions hybrid peer-review systems