“Fake news” has been in the news lately. It’s not just a phenomenon of your Facebook news stream, but also a problem in the academic publishing world.
As a scientist, I’m naturally intrigued about the latter and how fake research and sometimes completely fake papers are published. While researching for a blog post on the topic, I revisited Springer’s big fake paper scandal from 2014.
Scandals like that give a company the opportunity to show the public how they deal with problems and how sincere they are about their business. So it comes as a surprise that Springer published this press release a year after the incident making things — in my opinion — even worse.
The statement feels so surreal and bizarre that I decided to postpone my original planned post and just publish the text of the press release and add a few comments. After all, that’s what press releases are for and that was Springer’s intention when they published it.
The backstory
SciGen is a prank computer program that generates random computer science papers. Those fake papers look astonishingly real with interesting headings, graphs and citations. The program was developed by graduate students at MIT with the goal to bring attention to the problem of bogus conferences with low submission standards.
I remember when SciGen was first released. My colleagues and I played with it and it was great fun. However, SciGen is not fun for everyone…
In 2014, Springer, one of the biggest academic publishers, had to retract 18 papers from conference proceedings because they were generated by SciGen.
That’s embarrassing for a scientific publisher, very embarrassing indeed. I wondered how Springer went about addressing their “fake papers” problem. Compared to the fake news problem that became apparent during the 2016 US presidential campaign, Springer’s problem does not appear to be that hard: Just don’t publish random words that don’t make sense.
How not to fix things
A year after the scandal, Springer addressed the issue with the following press release. Here is what I think is wrong with this statement, point-by-point:
In February 2014, Springer learned that it had published 18 articles that were generated by the SCIgen computer program, which creates fake documents for submission as Computer Science and Engineering conference papers. Springer immediately reached out to Dr. Labbé, the leading expert in the field.
Assume the New York times published random gibberish in their paper. What would their reaction be? They probably would feel bad and apologize to their readers. Aware that there will always be errors, they even institutionalized their self-criticism with a blog about grammar and style errors: http://afterdeadline.blogs.nytimes.com. It’s obvious that they take pride in what they do.
So what does Springer take pride in as a scientific publisher? They published 18 articles consisting of random words under their name. Their reaction was simply “No problem, we immediately called the leading expert in the field of fake papers”. There is no indication of constructive self-criticism, which reiterates their earlier position in an initial response to the scandal from February 2014. Only in the final paragraph of their final report of the incident did they acknowledge the embarrassment and apologize.
After intensive collaboration with Dr. Cyril Labbé from Université Joseph Fourier in Grenoble, France, Springer announces the release of SciDetect, a new software program that automatically checks for fake scientific papers. […]
SciDetect scans Extensible Markup Language (XML) and Adobe Portable Document Format (PDF) files and compares them against a corpus of fake scientific papers. SciDetect indicates whether an entire document or its parts are genuine or not.
That’s the technical part of this post: SciGen uses a context-free-grammar based on user defined rules to generate random text. In other words, you input some typical jargon of a field and the program produces text according to these rules.
Springer’s new program basically reverses the process and checks if the text was produced by specific grammar rules of the fake paper generators. Although I’m not a computer scientist, it think it’s fair to say that solving this problem is not a scientific breakthrough in and of itself. At least not enough of one to warrant a press release… And stressing the fact that the program can read XML and PDF files does not make it more exciting.
I’m sure there are related computer science problems that would be fun to work on. Actually, I would have loved to see “intense collaboration” by a publisher and scientists solving some real problems. But from how the press release goes on, we see that was not the case at all:
The software reports suspicious activity by relying on sensitivity thresholds that can be easily adjusted. SciDetect is highly flexible and can be quickly customized to cope with new methods of automatically generating fake or random text. Springer uses the software in its production workflow to provide additional, fail-safe checking.
So it seems the goal was really to save them from future embarrassment by installing this program in their production workflow. That’s where it gets problematic. All it needs is one bored computer science student somewhere in the world writing a fake paper generator that cannot be picked up by this method and the whole thing falls apart. With SciDetect, Springer probably has made it easier to submit fake papers because they believe they can detect fake papers automatically.
“SciDetect, […], is a valuable building block for the future of academic publishing. […] explained Dr. Hubertus von Riedesel, Executive Vice President Physical Sciences and Engineering at Springer.
Scientific publishers don’t have it easy these days. Their value is questioned in a world transitioning from print to online journals and and their business model faces an increasing resistance by the scientific community. So some technical innovation in their field would definitely help their future.
In that light it seems bizarre that one of their executives publicly says that a computer program telling apart random words from a scientific article is “a valuable building block of the future of academic publishing”. To his defense, this statement might just be a “fake” quote that sounded good in the press release. It would be interesting to know if Dr. von Riedesel can repeat this sentence with a straight face.
It helps us ensure that unfair methods and quick cheats do not go unnoticed.
That’s another problematic statement because it suggests that programs like SciGen, the original fake paper generator, were developed to specifically to cheat and are therefore unethical. According to the final report of Springer the authors of the 18 fake papers indeed used it to increase their publication output. Of course, that’s a highly unethical use of SciGen.
But it seems Springer also take offense in the original point of SciGen which was to uncover huge quality problems in the academic publishing industry (from their first press release of February 2014):
There will always be individuals who try to undermine existing processes in order to prove a point or to benefit personally. Unfortunately, scientific publishing is not immune to fraud and mistakes, either.
It’s definitely legit to discuss the ethics of submitting a fake paper to a journal. With some effort I could write a completely fake story in my field of genomics to fool some editor and reviewers of some journal. That would be scientific fraud. But this is not what we are talking about here.
It’s about a scientific publisher that publishes a book of conference papers without having a human read it. The press release talks about “sensitivity thresholds”. But here’s the thing: You don’t need sensitivity thresholds if you hire someone with a basic understanding of the field who reads through a book before it goes to press. He or she she will spot SciGen generated papers with 100% sensitivity and 100% specificity.
To be fair, in their final report they also mention this most obvious way to address the problem: “In certain cases, and in certain disciplines, series editors and Springer in-house editors will inspect the actual manuscripts and their reviews […]”. Unfortunately, this statement can also be read as: In other cases and in other disciplines Springer will just print the proceedings and won’t bother reading them at all.
Also from the final report:
Although we are not looking to belittle the seriousness of our mistake, it should not be forgotten that only 18 out of more than eight million documents that Springer has published were produced by the SCIgen computer program.
Springer has published 8 million papers and only found 18 to be fake. That sounds like a reasonable error rate when publishing at Springer scale. The problem is, however, that it basically says only 18 people have submitted SciGen papers. It implies that probably thousands or even more papers have never been read by a human.
And to prove this point I think people should keep sending random gibberish to Springer conferences. Actually, Springer should welcome it the way tech firms invite “ethical hackers” to test their network.
What can be taken from all of this? Automatic detection of fake papers is not helpful, probably even counter-productive. If publishers take pride in their work they need to come up with other measures to ensure the quality of their publications (like having a human read them).
A better solution?
It’s easy to criticize problems and institutions without providing any alternative solutions. In this case, however, I think it would have been really straightforward to handle the situation. Obvious disclaimer: I’m not a publisher making 1.6 billion a year publishing academic literature so I might have missed something here. But what about that for a press release?
We are deeply embarrassed by the recent “fake paper” incident and want to apologize to the readers of our publications. It’s the worst case scenario for an academic publishing company and required radical measures.
We immediately stopped all publishing processes and implemented a new system in which each individual piece published under our name is signed off by an editor with knowledge of the subject. In cases where we don’t have the resources in house or cannot get sign-off by a verified and trusted external editor, we decided not to publish such publications any more. After review, we found many of them were of limited value for the academic community anyway.
One final note: I respect the academic work of Dr. Labbé and his students. They deserve lots of credit to uncover the problem in the first place. Who would have thought to mine the scientific literature for random papers? This post is about how the significance of this work and their application is interpreted by Springer, more specifically the PR team at Springer who released and signed off this press release.
