AI intensifies fight against ‘paper mills’ that churn out fake research

Jan 28, 2024

You can also search for this author in PubMed Google Scholar

You have full access to this article via your institution.

Artificial-intelligence tools that can generate realistic images and text are making it difficult to detect fake research.Credit: Westend61/Alamy

Advances in artificial intelligence (AI) are complicating publishers’ efforts to tackle the growing problem of paper mills — companies that produce fake scientific papers to order. Generative AI tools, including chatbots such as ChatGPT and image-generating software, provide new ways of producing paper-mill content, which could prove particularly difficult to detect. These were among the challenges discussed by research-integrity experts at a summit on 24 May, which focused on the paper-mill problem.

Paper-mill detector put to the test in push to stamp out fake science

"The capacity of paper mills to generate increasingly plausible raw data is just going to be skyrocketing with AI," says Jennifer Byrne, a molecular biologist and publication-integrity researcher at New South Wales Health Pathology and the University of Sydney in Australia.

"I have seen fake microscopy images that were just generated by AI," says Jana Christopher, an image-data-integrity analyst at the publisher FEBS Press in Heidelberg, Germany. But being able to prove beyond suspicion that images are AI-generated remains a challenge, she says.

Language-generating AI tools such as ChatGPT pose a similar problem. "As soon as you have something that can show that something's generated by ChatGPT, there’ll be some other tool to scramble that," says Christopher.

Anna Abalkina, a social scientist at the Free University of Berlin and an independent research-integrity analyst, suspects that there might be a delay in these AI tools becoming more apparent in the academic literature because of the length of the peer-review process. Perhaps in the next few months, "we will see the first stream of papers", she says.

Byrne, Christopher and Abalkina were participants at the UNITED2ACT summit last week, which was convened by the Committee on Publication Ethics (COPE), a non-profit organization focused on ethics in academic publishing based in Eastleigh, UK, and the International Association of Scientific, Technical and Medical Publishers (STM), based in Oxford. The summit brought together international researchers, including independent research-integrity analysts, as well as representatives from funding bodies and publishers.

The fight against fake-paper factories that churn out sham science

"It was the first time we had a group of people come together and co-create a set of actions which we’re going to take forward to combat this problem," says Deborah Kahn, a trustee of COPE and a research-integrity consultant, based in London. The group intends to publish its joint action plan soon.

When it comes to detecting paper-mill works, "there is absolutely an additional challenge which is posed by synthetic images, synthetic text, et cetera", says Joris van Rossum, programme director for STM Solutions, a subsidiary of STM. "There is a general realization that there is the potential of screening becoming more difficult," he says.

Kahn says that, although there will undoubtedly be positive uses of AI to support researchers writing papers, it will still be necessary to distinguish between legitimate papers written with AI and those that have been completely fabricated. "We have to really look at how we identify those things, and how we make sure that people have actually done the research. And there are various ways we can do that," she says.

One strategy discussed during the summit was to require authors to provide the raw data from experiments, potentially with digital watermarks that would enable publishers to confirm that those data are genuine.

Hundreds of gibberish papers still lurk in the scientific literature

Currently, requirements for submitting raw data vary significantly between publishers, says Christopher. Establishing a uniform set of requirements for the submission of raw data across publishers, taking into account differences between fields of research, could therefore be helpful, she says.

Sabina Alam, director of publishing ethics and integrity at Taylor & Francis, a publisher based in Abingdon, UK, agrees but says that such standards will take time to implement. "I can't imagine it being an overnight flip, because the reality is many institutions don't actually have the resources to offer data-management infrastructure," she says. "We don't want to penalize actual research."

The summit also discussed other strategies for tackling the problem of paper mills more broadly, including organizing an awareness day or week for researchers, as well as identifying ways for publishers to share relevant information on suspected paper mills — for example when publishers simultaneously receive submissions — without breaching data-protection rules.

STM is continuing to develop its own paper-mill detection software, while also collating resources on similar tools available elsewhere through its integrity hub. The apparent rise in paper mills increases demand for such techniques — both for detecting fake papers at the point of submission and for identifying those that are already published.

Taylor & Francis is among the publishers that are making use of such tools, and Alam says that a growing number of ethics cases — instances of potential misconduct that are flagged for further investigation — are being escalated to her team. Roughly half of these cases are because of paper mills, according to Alam. Her team saw the number of ethics cases increase more than tenfold from 2019 to 2022 — and so far this year, there have been almost as many cases as during the whole of 2022. "It seems to have been commercialized and scaled up," she says.

Nature 618, 222-223 (2023)

doi: https://doi.org/10.1038/d41586-023-01780-w

Paper-mill detector put to the test in push to stamp out fake science

‘Papermill alarm’ software flags potentially fake papers

The fight against fake-paper factories that churn out sham science

Hundreds of gibberish papers still lurk in the scientific literature

What makes an undercover science sleuth tick? Fake-paper detective speaks out

Publishers unite to tackle doctored images in research papers

China's research-misconduct rules target ‘paper mills’ that churn out fake studies

Why Nature will not allow the use of generative AI in images and video

Editorial 07 JUN 23

Preserve cultural diversity in author names

Correspondence 06 JUN 23

Tanzania's researchers offered US$22,000 to publish in international journals

News 06 JUN 23

Are you a postdoc working in academia or industry? Share your career experiences with Nature

Career News 07 JUN 23

How screenwriting can help your grant writing

Career Column 07 JUN 23

The surprising benefit of meditative walks

Career Column 07 JUN 23

Southern University of Science and Technology (SUSTech) is a public research university funded by Shenzhen city. Widely regarded as a pioneer and i...

Shenzhen

Southern University of Science and Technology (SUSTech)

Chinese Academy of Sciences, seeks exceptional, creative scientists to join its faculty.

Shanghai, China

The Institute of Neuroscience,Center for Excellence in Brain Science and Intelligence Technology,CAS

Situated in the historical and cultural city of Nanjing, CPU seeks talented scientists from the globe.

Nanjing, Jiangsu, China

China Pharmaceutical University

Summary A postdoctoral position is immediately available in the laboratory of Dr. Jun Yang in the Department of Surgery, St. Jude Children's Resear...

Memphis, Tennessee

St. Jude Children's Research Hospital (St. Jude)

Texas

The University of Texas Health Science Center at San Antonio (UT Health Science Center at San Antonio)