The desk rejection rate at Transactions on Machine Learning Research sat around 6 percent in 2023. This year it crossed 53 percent, according to TMLR Editor-in-Chief Nihar Shah's account of a recent experiment. That spike tells you something is flooding in, but it does not tell you what. So during his August 14 to 28 rotation, Shah picked 10 papers slated for desk rejection and asked their authors to get on a Zoom call and explain them.
The results paint a grim picture of what peer review is now catching. Of 10 papers, the authors of three could not answer basic questions about their own submissions. Three more could handle high-level concepts but stumbled on technical details. One author who answered everything had a major error in a main claim. Two authors who struggled in the meeting later sent follow-up emails that an AI detection tool flagged as fully AI-generated. One author accidentally described a p-hacking workflow when trying to explain their methods.
For builders who rely on published research to make architecture and investment decisions, this is a reminder that the paper pipeline has a quality problem growing faster than anyone's ability to filter it.
What did the TMLR editor actually do?
Shah's experiment was straightforward in design but expensive in practice. During his two-week rotation as Editor-in-Chief, he identified 10 papers from the desk rejection queue. Instead of rejecting them, he sent each author team a message through OpenReview asking for a 30-minute meeting to discuss the paper before a decision was made.
The response itself was a data point. One team withdrew immediately. Another said they were too busy. Shah scheduled meetings with the remaining eight. One author did not show up. That left seven actual conversations over roughly 20 to 25 hours of work across two weeks, on top of his regular editorial duties.
In each meeting, Shah asked two categories of questions: basic ones about the problem setting, notation, and results claimed, and deeper ones about specific technical expressions, theoretical results, and experimental design choices. He used LLMs as an aid to understand the papers himself, particularly for concepts outside his expertise.
The author pool was diverse in credentials: undergraduates, master's students, PhD students, faculty, and independent researchers. Most but not all papers were solo-authored. The credential mix matters because it shows the problem extends beyond students. Faculty and independent researchers were in the group too.
The chart above breaks down all 10 submissions by outcome. Three papers had authors who could not answer basic questions, and all three were solo-authored. Two of those authors appeared to have almost no substantive understanding of their own papers. The third could not locate where key results claimed in the abstract were actually presented in the document.
Three more papers had authors who handled the basics but faltered on deeper questions about technical aspects or design choices. One paper's authors answered everything, but Shah found a major error in one of the paper's central claims, which the authors acknowledged.
All 10 papers were ultimately desk rejected. Only the one with the answered questions received an invitation to resubmit.
Why is the desk rejection rate climbing so fast?
TMLR's desk rejection rate went from roughly 6 percent in 2023 to about 53 percent today. That is a structural shift in what the venue is receiving.
The surge in submissions is driven partly by the same forces that made ChatGPT a household name. Large language models can produce paper-shaped documents in minutes. Whether those documents contain correct math, valid experiments, or genuine insight is a separate question, and it is the question Shah was trying to answer.
Ehud Reiter, a researcher who has written about the review crisis, argues that desk rejection is now an "unfortunate necessity" given the volume of low-quality and LLM-assisted submissions flooding journals. TMLR has responded with submission quotas aligned to recent patterns and an emphasis on clear writing, as Shah notes in his account.
The problem extends beyond TMLR. Conferences like NeurIPS and ICML have reported similar surges. The review infrastructure, built on volunteer labor, was not designed for a world where generating a plausible-looking paper costs approximately zero.
For a sense of the scale: TMLR's entire editorial team, from reviewers to Action Editors to Editors-in-Chief, consists of unpaid volunteers. When the submission pipeline multiplies, the volunteer pipeline does not. Desk rejection is the pressure valve.
How many authors actually understood their papers?
This is where the numbers get uncomfortable.
Of the seven authors who showed up to meetings, only one could answer all questions about their own paper. That is roughly 14 percent of the participating authors. Even that paper had a major flaw.
The three who could not answer basic questions are the most striking cases. These were not trick questions. Shah asked about the problem setting, the notation, and where results were presented. Two authors had near-zero understanding of the contents. One could not point to where their own abstract's key claims were supported in the paper.
The follow-up emails add another layer. Two authors who struggled in the meeting sent written responses afterward. Shah ran those emails through Pangram, an AI text detection tool, which classified both as "100% AI." This does not prove the papers themselves were AI-generated, but it is a strong signal that the authors' attempted explanations were.
Then there is the p-hacking anecdote. One author, asked to describe their analysis methods, ended up describing an entire workflow for manufacturing statistical significance. Not on purpose. They simply described what they did, which was p-hacking.
Gautam Kamath, a researcher who commented on the experiment on Bluesky, summarized the finding bluntly: people submitting these papers have "no idea" what is going on in them.
What does this mean for builders who rely on research?
If you are an engineer, founder, or product lead making decisions based on published ML research, the implication is direct: the paper pipeline is a weaker signal of vetted work than it was two years ago, and you need to filter harder.
- Your literature review process needs upgrading. A paper that appears in a venue's proceedings is weaker evidence than it was in 2023. Check who reviewed it, how many reviewers were involved, and whether the venue has a desk rejection process that actually screens for AI-generated noise.
- Reproduction is now mandatory, not optional. If a paper claims a result you want to build on, reproduce the core experiment before committing engineering time. This was always good practice. It is now survival practice.
- Author credibility is a weaker signal. A faculty byline no longer guarantees the faculty member wrote or understood the paper. Check whether the authors have a track record of related work, not just a single suspicious submission.
- The cost of bad research is compounding. If you implement a method from a paper that no one, including the authors, actually understands, you are building on sand. The debugging cost alone can exceed the cost of a more conservative approach.
This connects to a broader pattern we have tracked at Data Today. When we looked at how frontier LLMs can detect when they are being tested, the finding was that evaluation itself is under pressure from the systems being evaluated. Here, the evaluation infrastructure for human research is under pressure from AI-generated submissions. The gatekeepers and the generators are in an arms race, and the gatekeepers are losing ground.
Can peer review scale to catch AI-generated submissions?
Shah's experiment was illuminating but it does not scale. He spent 20 to 25 hours over two weeks to evaluate eight papers. TMLR receives far more than eight papers per two-week window. The math does not work for venue-wide implementation.
Shah acknowledged this directly, noting that the process "seems hard to scale, especially given the large numbers of submissions." He pointed to ongoing work at CMU on more scalable approaches to author verification, including a system called greCAPTCHA that aims to test whether authors understand their own submissions at scale.
The core tension remains. Peer review is a volunteer system. AI paper generation is an automated system. The asymmetry favors the generator. A single person with an API key can produce dozens of paper-shaped documents per day. A reviewer needs hours to properly evaluate one.
The desk rejection rate of 53 percent is TMLR's current answer to this asymmetry: filter aggressively at the top, before sending papers to reviewers. The question is whether 53 percent is high enough, and whether the filter is catching the right things.
Shah's experiment suggests the filter is working in one sense: the papers it flags are indeed weak. Seven out of seven authors who showed up had significant problems with their papers. That is a 100 percent hit rate for the desk rejection heuristic. The concern is what slips through: papers that are AI-generated but polished enough to pass desk review, and then consume reviewer time or even get accepted.
What should change about how we evaluate research?
TMLR is already adapting. Shah mentions several initiatives the venue is running alongside the desk rejection ramp: submission quotas, emphasis on clear writing, and experiments like this one. Other venues are watching.
For the broader research community, the experiment raises questions that go beyond logistics.
If authors cannot explain their papers, what does authorship mean? The current system assigns credit through authorship. If authorship no longer implies understanding, the credit signal is broken.
If authors who cannot explain their papers are later asked to review others' papers, the review system degrades further. TMLR and similar venues recruit reviewers from the author pool. Contaminating that pool with people who do not understand their own work is a slow-acting poison.
If AI detection tools like Pangram are being used to flag follow-up emails, should they be used earlier in the pipeline? The answer is not straightforward. Detection tools have false positives and false negatives, and adversarial actors can adapt. But the signal from Shah's experiment, where two follow-up emails scored as fully AI-generated, suggests these tools have at least diagnostic value.
The most defensible near-term change is what TMLR is already doing: raise the desk rejection bar, require clear writing, and run periodic audits. The less defensible position is pretending the submission volume is a normal fluctuation that will subside on its own.
The signal is the problem
Nihar Shah spent 20 hours to confirm what many suspected: a meaningful fraction of papers now submitted to ML venues are not understood by the people who put their names on them. The desk rejection rate at TMLR went from 6 percent to 53 percent in under three years. The filter is working harder than ever, and the flood is still rising.
For anyone building on published research, the practical response is to keep reading papers while treating the publication itself as a weaker signal. Read the paper, reproduce the result, check the author's track record, and treat every claim as a hypothesis until your own code confirms it. The venue name on the header is worth less than it was last year. Next year it will be worth less still.
Sources
- medium.com (@TmlrOrg) , Nihar Shah, "Asking Authors About Their Own Papers," TMLR, September 2026
- ehudreiter.com , Ehud Reiter, "Desk rejection is an unfortunate necessity," September 2026
- bsky.ahri.cloud (gautamkamath.com) , Gautam Kamath, Bluesky post on the TMLR experiment, September 2026
- jmlr.org , Transactions on Machine Learning Research, official site
- jmlr.org , TMLR Action Editor guide
- cs.cmu.edu , Nihar Shah et al., greCAPTCHA preprint, CMU
