The rise of large language models is introducing fabricated references and unverifiable data into scientific literature, forcing physicists to scrutinize sources and reinforce core research skills to maintain trust in published results
As artificial intelligence systems capable of generating human-like text become widely accessible, the scientific community faces a growing challenge: distinguishing reliable research from fabricated or misleading content. Large language models (LLMs), which underpin many modern chatbots and automated writing tools, can produce plausible-sounding scientific articles, references, and even entire research narratives that may not correspond to any real experiment or publication. This development is undermining the traditional mechanisms by which physicists and other researchers verify the provenance and accuracy of scientific claims.
Historically, the process of scientific publication relied on the traceability of ideas to identifiable authors, institutions, and experimental results. When a paper cited another, it was generally possible to locate the referenced work, assess its methodology, and evaluate its credibility. However, the proliferation of AI-generated content has introduced a new layer of uncertainty. Researchers now encounter references that appear legitimate but may be entirely fictitious, or data that is superficially convincing yet lacks any underlying measurement or experiment.
Fabricated References and Policy Response
Recent incidents have highlighted the scale of the problem. In 2024, a research team deliberately created a fictitious medical condition and disseminated it through preprints and online posts to test whether AI systems would propagate the false information. The experiment demonstrated that LLM-powered tools can amplify fabricated content, including references to non-existent studies and acknowledgments of fictional institutions. Even when the original text contained clear signals of fabrication, such as overtly fictional affiliations or explicit disclaimers, AI systems reproduced and spread the material as if it were genuine.
Physics publishing platforms have responded with new policies. For example, arXiv, a major preprint repository, announced in May that submissions containing incontrovertible evidence of unchecked LLM-generated content-such as hallucinated references-would result in a one-year ban for the authors. After such a ban, future submissions would require prior acceptance at a reputable peer-reviewed venue. These measures are intended to deter careless or deceptive use of AI tools, but they also raise the risk that well-intentioned researchers could be penalized for inadvertently citing fabricated material.
Risks for Physicists and Research Integrity
The risk is not limited to those who use LLMs directly. As the information ecosystem becomes saturated with AI-generated slop, even careful researchers may unknowingly incorporate fabricated references into their work. A common shortcut-copying a citation from a trusted paper without verifying the original source-can now lead to the propagation of non-existent studies. In one documented case, a peer-reviewed article was retracted after it was found to cite a preprint describing a fabricated disease, despite the authors' objections. Such incidents expose all co-authors to reputational and professional consequences, including bans from major repositories.
To maintain the integrity of scientific literature, physicists must reinforce the skills that underpin rigorous research: critical evaluation of sources, information literacy, transparent communication, and careful scholarship. These are precisely the competencies that AI companies often claim their systems can replicate or replace. However, as the reliability of the scientific record comes under threat, the value of human expertise and judgment becomes more apparent. Avoiding the use of LLMs in writing is no longer sufficient; researchers must actively verify every reference and claim, regardless of its apparent plausibility.
Slowing Down to Preserve Trust
The current environment demands a more deliberate approach to scientific writing and review. What once seemed like solid ground-trusted journals, familiar authors, established institutions-may now conceal fabricated or unverifiable content. Moving safely through this landscape requires slowing down, double-checking sources, and building new tools and practices for verifying information. The process is labor-intensive and may reduce the speed of publication, but it is essential for preserving the credibility of physics research and ensuring that future advances are built on a reliable foundation.
One central concept in this discussion is the role of verification in scientific publishing. Verification involves tracing each claim, reference, or dataset back to its original source, confirming that the underlying experiment or analysis was actually performed, and assessing the quality of the evidence. In the context of quantum physics and experimental science, this means not only checking that a cited paper exists, but also that its measurements, device specifications, and reported uncertainties are credible and reproducible. As AI-generated content becomes more prevalent, robust verification practices are critical for distinguishing genuine advances from artifacts of automated text generation.
