Gary King and Steven Worthington of Harvard's Institute for Quantitative Social Science compared AI with the strongest human reviewer panel in a preprint released on October 22, and the result was one-sided: Claude won on all seven review dimensions.
The study used 124 papers and 134 paid expert reviews from Harvard's rapidPeer service from 2019 to 2026 as the human baseline. Claude Opus 4.8 competed at its maximum reasoning setting, with one group also allowed to search the web, averaging 12.4 searches per paper. Judging came from blind pairwise comparisons by OpenAI's GPT-5.5 and from three former journal editors who did not know which review was machine-written.
- Higher scores across the board: AI averaged 8.7 points against 6.0 for humans; the former editors chose the AI review in 26 of 30 blind judgments, with the largest gaps on sourcing and on figures and tables.
- More accurate, too: checking every claim against the manuscript, the human error rate was 15.9%, against 5.5% for Claude; none of the 651 citations carrying a DOI or arXiv identifier was fabricated.
- A proposal, and a warning: the authors suggest journals publish their AI review prompts and let authors pre-review with AI before submission, while the final publication decision stays with human editors; they also warn that AI assistance could make scholarly ideas converge.
Original link: Click to view the original
Comments