Startup Investigation Uncovers 50 Peer-Reviewed Papers with Fake Citations

A recent investigation by the startup GPTZero has revealed that 50 peer-reviewed papers submitted to the International Conference on Learning Representations (ICLR) contained at least one fabricated citation, a phenomenon known as “hallucination.” This term refers to instances where artificial intelligence (AI) generates fictitious references. ICLR is recognized as a leading academic conference in the field of deep learning.

The investigation, conducted by three researchers based in Toronto, utilized GPTZero’s Hallucination Check tool to analyze 300 submissions to the conference. Findings indicated that 50 of these papers featured at least one “obvious” hallucination, despite being reviewed by multiple experts. Many of the citations were attributed to non-existent authors or incorrectly linked to journals, and some had no valid counterpart at all.

According to the report, had there been no intervention, these papers might have received high enough ratings to be published. Alex Cui, co-founder and Chief Technology Officer of GPTZero, expressed surprise at the results, stating, “We just struck gold but kind of in the wrong way. There’s probably a lot more where that came from.”

In collaboration with ICLR program chairs, the authors of the investigation are working to identify additional papers with fabricated citations. Cui noted that they are currently examining all 20,000 articles submitted for ICLR 2026, with a deadline approaching for acceptance announcements. “We’re under a bit of a time crunch, but I think we can get it done,” he affirmed.

Colin Raffel, an associate professor at the University of Toronto and a program chair at ICLR, assured that he and his team are actively rejecting submissions that breach the conference’s policies. GPTZero also plans to expand the use of its detection tool to other conferences and apply its model to additional scientific reviews.

Founded by Cui and Edward Tian, GPTZero began as a web application in December 2022 and rapidly gained traction, attracting 30,000 users. Following its official launch in January 2023, the user base surged to 4 million in 2024, culminating in a $10 million preemptive Series A funding round led by Footwork co-founder Nikhil Basu Trivedi. Currently, the company claims to have approximately 10 million users, including institutions such as Purdue University and UCLA.

Blair Attard-Frost, an assistant professor of political science at the University of Alberta and a fellow at the Alberta Machine Intelligence Institute, highlighted the implications of GPTZero’s findings within the context of growing AI usage in academia. “You have this giant influx of even more papers that puts more strain on peer-review processes,” she said. Many academics are already overwhelmed and may lack the capacity to thoroughly review submissions.

The conversation around AI’s role in academia has gained momentum, especially regarding its application in both writing and reviewing academic papers. A study published in September 2023 in the Yale Journal of Biology and Medicine praised OpenAI’s ChatGPT for its potential in the journal review process, noting its ability to identify methodological flaws.

Yet, further research has underscored significant limitations in AI’s capability to accurately reference citations. One study indicated that LLMs reviewing submissions for ICLR 2024 inflated authors’ scores and increased acceptance rates for papers. Another report, published in April 2023 by Kevin Wu et al., found that between 50 percent and 90 percent of LLM-generated responses were either poorly supported or contradicted the sources cited.

The issue of false citations extends beyond academia. In November, a report commissioned by the Newfoundland and Labrador government and prepared by Deloitte was found to contain incorrect citations likely generated by AI. Following this revelation, the government mandated a review of AI guidelines for commissioned reports. Deloitte has also faced scrutiny for including citations of non-existent academic papers in prior government reports in Australia.

Earlier this year, Canada’s Minister of AI and Digital Innovation, Evan Solomon, emphasized that economic benefits of AI should take precedence over stringent regulation. Since then, Solomon has hinted at potential legislation aimed at regulating deepfakes and data privacy.

Attard-Frost expressed skepticism about the federal government’s approach, asserting that academic professionals should not rely solely on government intervention. “Peer review is essentially free service work,” she noted, highlighting the need for better support for academic workers.

Cui emphasized the importance of proper implementation of AI in academic contexts, stating, “There’s a right and a wrong way to do it.” He advocated for using AI responsibly while ensuring that outputs meet rigorous standards. Attard-Frost raised concerns about GPTZero’s claimed 99 percent accuracy in detecting falsified citations, suggesting that such a high rate could lead to numerous false positives among ICLR’s submissions, potentially damaging the reputations of authors whose work is mistakenly flagged.

To mitigate issues related to LLM-generated false citations, Attard-Frost proposed solutions such as a “compounding fee submission model,” which would charge authors escalating fees for multiple submissions and an “endorsement model,” validating submissions through verified individuals. She stressed that there is no one-size-fits-all solution, indicating that responses must be tailored to specific applications.

Cui remains hopeful, asserting that public accountability can foster improvements in addressing the rise of false citations. “It’s not like a lost cause; we actually can create tools to do this,” he concluded.