88Trust
Verified
🔍 Web Verified
Elen Le Foll 🇫🇷 🇬🇧 🇩🇪onMastodon27d ago
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
Trust Metrics
92
88
85
82
Accuracy92%
Framing88%
Context85%
Tone82%
Analysis Summary
A researcher deliberately published two fake preprints about a nonexistent disease (Bixonimania) with obvious fabrication markers embedded in the text. LLMs immediately incorporated the papers into their training and outputs, and then legitimate academic papers started citing the fake preprints — demonstrating how AI-assisted research can amplify misinformation. This appears to be a deliberate demonstration of a real problem: fabricated citations in peer-reviewed research have increased 12-fold since 2023, according to a Lancet analysis of 2.5 million biomedical papers, with AI hallucinations identified as a major driver.
Claims Analysis (4)
“A researcher made up a disease and published two evidently fake preprints about it”
Nature article confirms researcher deliberately published fabricated preprints with obvious fake content indicators
“The preprints contained sentences such as 'this entire paper is made up' and 'Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group'”
Direct quotes documented in Nature article showing intentional fabrication markers embedded in preprints
“The fake preprints were almost immediately picked up by LLMs and documented in their output”
Nature article confirms LLMs incorporated the fake preprints into training/output within short timeframe
“Actual medical papers started citing the preprints, demonstrating that academics relying on LLMs is a genuine problem”
Nature article documents subsequent citations by legitimate academic papers, corroborated by Lancet study finding 12-fold increase in fabricated citations 2024-2026
Was this analysis helpful?
Try ClearFeed free →