A study of 50K peer reviews for CS articles published in AI conference proceedings in 2023 and 2024: 7%-17% of the sentences in the reviews were written by LLMs
Context & Ripple Effects
Earlier publisher policy drew a line between AI assistance and authorship, permitting help with writing only when disclosed. This study extends the governance problem from manuscripts to the evaluations that determine which work is accepted.
The finding foreshadows later reports of substantial AI use in ICLR reviewing and the subsequent push by conference organizers to restrict LLMs in writing and review workflows, including reported AI-generated ICLR reviews and conference restrictions on LLM use.
First-order effects
- Program chairs and authors gain evidence that peer-review text in recent AI proceedings was already partly produced by LLMs, making reviewer disclosure and review-quality checks more consequential.
- The result complicates the signal conveyed by a review: readers of reviewer feedback cannot automatically distinguish a reviewer’s own assessment from model-assisted prose.
Second-order effects
- Conference organizers face pressure to define permitted assistance, disclosure requirements, and enforcement methods; blanket bans and permissive policies create different compliance burdens.
- As AI assistance can raise research output—as later analysis associated apparent LLM use with higher arXiv posting rates—review capacity may become a tighter constraint if submission volume rises faster than credible human evaluation.
Third-order effects
- Scientific publishing may move from treating AI authorship as the central issue to governing provenance across the full research workflow, especially peer review.
- If AI-written reviews persist, conferences will need to balance scalable review assistance against preserving accountable expert judgment; the corpus does not establish which policy approach will prove effective.
The trend: Generative AI is shifting scholarly publishing’s core challenge from authorship attribution toward verification, disclosure, and accountability across both papers and peer review.