Pangram Labs: ~21% of the 75,800 peer reviews submitted for ICLR 2026, a major ML conference, were fully AI-generated, and 50%+ contained signs of AI use

By - Miryam Naddaf 0 — Miryam Naddaf is a science writer based in London. — Search author on: — PubMed Google Scholar

Nature 2025-11-30 Miryam Naddaf

Context & Ripple Effects

This is a sharp escalation from an earlier analysis that found LLM-written material in portions of computer-science peer reviews, rather than wholesale reviews: a study of 50,000 reviews found AI-written sentences in reviews. The reported ICLR scale turns AI use in evaluation from an isolated authorship question into a conference-operations problem.

It also sits alongside a subsequent push by AI conferences to restrict LLM use in paper writing and reviewing, while the later scrutiny of Pangram’s false-positive claims underscores that detection cannot by itself settle misconduct cases.

First-order effects

ICLR’s review process faces a more immediate credibility and quality-control burden: a sizable share of submitted assessments may not represent independent reviewer judgment.
Pangram’s findings give conference organizers a concrete basis to tighten disclosure, review, and enforcement rules around AI-assisted evaluations.

Second-order effects

Review platforms and program committees will be pushed to add audit trails, declarations, or targeted checks, increasing administrative work for reviewers and organizers.
Detection results become consequential inputs to academic decisions, raising pressure for processes that distinguish prohibited automation from permitted assistance rather than treating a detector score as dispositive.

Third-order effects

If AI-generated reviewing remains widespread, peer review may shift toward governance systems that verify reviewer contribution and accountability, not simply anonymous text submission.
The durable tension will be between scaling review with AI tools and preserving trusted human evaluation; restrictive policies may spread, but their legitimacy will depend on reliable appeals and evidence standards.

The trend: Generative AI is moving from assisting research workflows into governing them, forcing scholarly institutions to operationalize rules for provenance, disclosure, and accountability.

Discussion

@southernwintrs Will I Am on x
A paper that is edited by AI is not necessarily bad! I don't think my papers, which I edit with AI, is made worse by the fact that I use it as a tool (where I supervise and edit inputs and outputs). Kinda funny that at AI conferences, using AI tools is now considered *bad*.
@v_shaal Vishal Verma on x
The real question: what's worse - AI-generated reviews of real research, or human reviews of AI-generated papers? Because if both sides are automating this, we've basically just created an extremely expensive bot-to-bot book club.
@b_shrir Sriram B on x
Just tried this for the papers on which I am a reviewer. For many of them, my review is the only human-generated one😂
@pandaashwinee @pandaashwinee on x
it would be interesting to understand whether the AI reviewers prefer AI writing. my understanding so far is that they do, but i wonder if anyone has looked into this quantitatively?
@damienteney Damien Teney on x
Amazingly the set of people complaining about low-quality reviews/papers seems completely disjoint from those who wrote low-quality reviews/papers. 🤔
@nogarot Noga H. Rotman on x
Super interesting findings, especially on the submissions. Most papers seem to have very little AI-generated content, and lower AI usage actually correlates with higher scores. Some encouraging data in all of this.
@lovvge Bernie Wang on x
Both as an AC and author, i couldn't care less who / what writes the reviews ... the only thing I care about is whether, underneath the tokens, there are valid points.
@dawei_li_asu Dawei Li on x
With ~21% of #ICLR reviews flagged as AI-generated, maybe it's time to ask: who's judging your paper — a human or an LLM? In our recent paper: “Who's Your Judge? On the Detectability of LLM-Generated Judgments” we study how to detect LLM-generated judgments and reviews — even [im…
@abhinav95_ Abhinav Shukla on x
Good news: this tool detects all my ICLR reviews as fully-human written. Bad news: a lot of reviews on the papers in my stack are fully AI generated (including one in which mine is the only one detected as human-written!).
@dcml0714 Cheng Han Chiang on x
Glad to see that all four reviewers I spent a lot of time writing for are classified as full human-written. Kind of
@canyuchen3 @canyuchen3 on x
🚨21% of ICLR 2026 reviews are fully AI-generated, and 35% are AI-edited🚨. We are already living in an era where AI silently shapes scientific peer review. Understanding, detecting, and defining 𝐡𝐮𝐦𝐚𝐧-𝐀𝐈 𝐜𝐨-𝐚𝐮𝐭𝐡𝐨𝐫𝐬𝐡𝐢𝐩 is no longer optional, but is critical [image]
@alexolegimas Alex Imas on x
More than a 5th of reviews at a top CS conference were fully AI-generated. These reviews were materially different both in terms of content and scores assigned. Curious what this looks like for Economics. Easy to do, and I'm happy to facilitate if any editors want to run this.
@shuaichenchang Shuaichen Chang on x
Now we're trusting AI to detect AI-generated content more than we trust AI-generated content itself haha. To be clear, I have full respect for the people who put in the effort to make this analysis possible. From my own experience: I have one submission where all four reviews
@xianjun_agi Xianjun Yang on x
It's worth noting that most AI detectors can only return a probability of whether the text is AI-generated. But a high probability alone can not serve as practical evidence. Luckily, our previous work published at ICLR 2024 can provide strong text-level EVIDENCE to support the
@max_spero_ Max Spero on x
Curious about AI use in paper writing or reviews? We ran every paper and every review through @pangramlabs, and this is what we found. 🧵
@danielkhashabi Daniel Khashabi on x
Fascinating snapshot of AI's impact in scientific writing/reviewing from this ICLR cycle: (1) More AI in the writing correlates with *lower* review scores. (2) More AI in the reviews correlates with *higher* scores. Lots of room to improve our tech tools (and our habits!)
@mbodhisattwa Bodhisattwa Majumder on x
Super interesting! Though I'd be wary of false positives & cases of AI editing when disclosed. E.g., one of my reviews is tagged as “lightly AI-edited,” which is, of course, wrong. :) Though I think @pangramlabs's false positives for fully AI-generation are very, very low.
@dogacel0 @dogacel0 on x
I wanted to uncover some interactions in ICLR data on AI-usage in submissions and reviews, so I analyzed it further. What surprised me is that even the fully AI reviews gave lower scores to submissions with more AI-generated content on average. AI still prefers human-written [ima…
@leonpalafox Leon Palafox on x
@gneubig @pangramlabs I'm afraid this may flag people who are not as fluent in English and just wanted it to help them correct grammar
@kahnchana Kanchana Ranasinghe on x
@gneubig @pangramlabs Interesting! One paper I reviewed (which I actually liked) was withdrawn after getting two bad review scores (from other reviewers). This analysis tags two of those reviews as 100% AI generated.
@gneubig Graham Neubig on x
Obvious caveat: LLM-generated text detection is not perfect so there will be mistakes. Take this as a guide, not as the truth. But the reason why I asked for this was because I suspected several of my reviews were AI generated, and the results matched w/ my intuition.
@shuaichenchang Shuaichen Chang on x
@gneubig @pangramlabs From my own experience: I have one submission where all four reviews were flagged as “Fully AI-generated.” Interestingly, I actually found the reviews quite reasonable, and we're addressing them seriously. I've also reviewed five papers myself. I write the r…
@max_spero_ Max Spero on x
@gneubig @pangramlabs Thanks for the push to do this. People deserve more transparency around AI use. Nobody wants to spend time digging into a nonsensical paper that is largely AI-generated. And nobody wants to spend hours on new experiments to respond to a review that came stra…
@xing_rui12683 @xing_rui12683 on x
I check my reviews. Result in 2 moderate AI edited, 2 heavily and 1 light. Because I wrote review in Chinese and gpt helps me to translate into English. It is not a surprising result. But I think i am a responsible Reviewer :)
@haryoaw @haryoaw on x
Should we add a feature to OpenReview to show the confidence percentage of a paper and its reviews, indicating whether they are AI-generated?
@max_spero_ Max Spero on x
We were curious about our false positive rate, so we ran all ICLR 2022 reviews (pre-ChatGPT) as a baseline. Lightly AI-edited FPR: 1 in 1,000 Moderately AI-edited FPR: 1 in 5,000 Heavily AI-edited FPR: 1 in 10,000 Fully AI-generated: No false positives [image]
@gneubig Graham Neubig on x
ICLR authors, want to check if your reviews are likely AI generated? ICLR reviewers, want to check if your paper is likely AI generated? Here are AI detection results for every ICLR paper and review from @pangramlabs! It seems that ~21% of reviews may be AI? [image]
@carnage4life Dare Obasanjo on bluesky
The enshitification of scientific publishing marches on. First arXiv had to clamp down on submissions thanks to a flood of AI-written papers. — Now ICLR authors say their peer reviews were churned out by AI. Reviews full of hallucinated content that didn't exist and offering …
@drjacekdebiec @drjacekdebiec on bluesky
Why should universities keep academics who, instead of writing their papers, reviews, etc., use AI to do it for them, #AcademicSky? — ‘Writing is an integral part of the process of understanding.’ — Hannah Arendt — www.nature.com/articles/d41...
@katjadiehl Katja Diehl on bluesky
It's a slop. — “Major AI conference flooded with peer reviews written by AI. — Controversy has erupted after 21% of manuscript reviews for an international AI conference were found to be generated by artificial intelligence. I don't want mobility to be planned by AI. — www…
@elfsternberg Elf M. Sternberg on bluesky
Conference on AI flooded with papers and reviews written by AI. Why have humans go at all? www.nature.com/articles/d41...
r/nottheonion r on reddit
Major AI conference flooded with peer reviews written fully by AI
r/Futurology r on reddit
Major AI conference flooded with peer reviews written fully by AI
r/singularity r on reddit
Nature: Major AI conference flooded with peer reviews written fully by AI
r/technology r on reddit
Major AI conference flooded with peer reviews written fully by AI
r/technews r on reddit
Major AI conference flooded with peer reviews written fully by AI

Chronicles