Teaching in the Beginning

Over a decade ago, I participated in a meeting on the ethical foundations of reviewing interdisciplinary science in the framework of European COST Action directed at the next generation of young scientists. I vividly recall my shock when, for the very first time, I heard about some startling cases of fraud and misconduct relating to the established procedures for publishing in peer-reviewed scientific journals. I could hardly have imagined then how proudly I was to carry the title of sleuth.

Prompted by the same ethical values underlying that framework, I have since been informing doctoral students of malpractice in scientific publishing as part of their training. Raising awareness of this form of malpractice was my only activity in the public arena for some years, until I started to use X (formerly Twitter) for sharing the same sort of information in Spanish, gathering a small number of followers. Privately, I obtained a trainer certification in research ethics and integrity before starting to follow the work of other sleuths.

In all likelihood, the story could have ended there, were it not for a chance discovery. It is curious how events in life that might at first appear irrelevant can subsequently become inflection points that determine the future. I say as much because, in retrospect, that discovery was an event that changed many aspects of my life, from professional relationships to working methods. I had acquired the habit of combing through open peer reviews from time to time, whenever they were easily accessible. On occasion, I noted features that I would post on my X account called #ReviewLikeATweet.

Identifying the First of Many Review Mills

One day, early in the summer of 2023, I came across a spelling mistake that awakened my curiosity: “The conclusion part is very week. Improve by adding the results of your studies”. I wondered whether anyone else had made that same spelling mistake, as it appeared odd. After a simple Google search, I unearthed a good number of identical phrases with the same mistake in other reviewer reports. I must confess I was flabbergasted. But what was even more astounding was that both the content and the structure of all the reports were practically identical, even though they had ostensibly been written by different individuals in relation to a wide range of scientific papers published in a broad spectrum of scientific journals.

Without exception, in the introduction of each review report was the request for the (coercive) citation of articles, written either by the reviewer themselves or by other authors — whose names I recognized as the reviewers of the other reports turning up in my searches. How could that be? How could ten different researchers, all affiliates of various academic institutions, repeat the same comments in 84 reports and request on each occasion that their own works be cited?

I understood it was highly likely that the reviewers were working in coordination. Having overcome my initial feelings of astoundment and even incredulity in the face of such a blatant disregard of reviewer ethics, I initiated an investigative phase, diving into databases and trying to find literature on the Internet that might explain these findings. I was trying to find literature referring to such documents — template review reports with boilerplate comments and coercing citations from authors — but was unsuccessful. This appeared to be a new phenomenon.

X was the only platform I was using at that time, and given its structure, the use of screenshots appeared to be the most reasonable way to summarize my findings. My closest friends advised me to let it go and not go public, fearing retaliation against me. I also doubted whether it would be convenient to publicize the findings within the academic community through PubPeer, a platform on which I had seen the work of other sleuths, as I thought at the time that this was an isolated case, devoid of much interest.

Curiously, it was more difficult for me to decide on using PubPeer than to choose between either an anonymous or a signed PubPeer login, perhaps because I do not use an anonymous profile on X. In any event, I needed a hashtag on X: creativity not being one of my defining characteristics, I decided to see whether someone among my (at that time) limited number of followers could give me an idea. Very kindly, they came up with some suggestions, and #ReviewMills was chosen, due to its implicit notion of mass production.

The Problem at Scale

As I published new cases every week, more people expressed interest in the problem, encouraging me to continue. I gratefully received their valuable advice to save all the web pages (box). What I could never have imagined was that the case that I had started to make known in October 2023, which I considered one of a kind, would turn out to be nothing other than the first of over 80 review mills that have been revealed to date, covering more than 1000 reviewer reports from 100+ named reviewers (together with an indeterminate number of anonymous reviewers).

Recently, working alongside Dorothy Bishop and René Aquarius, we published a study on the specific case of a review mill in obstetrics and gynecology, the size and extent of which was immense: 195 review reports and 9 named reviewers identified, in addition to various anonymous reviewers. The number of verified Web-of-Science peer reviews was well over 3000, and that solely by the 9 named reviewers operating in the review mill in medicine. I have identified many other review mills, which I will gradually be placing in the public domain; it appears that we have only begun to scratch the surface of this serious problem.

See a documented review mill case on PubPeer

Such a rapid accumulation of so many peer reviews confers a somewhat implausible level of prestige upon a reviewer. But contrary to what they might appear to be, rather than pecata minuta, review mills have serious repercussions for academia, science, and society beyond the fake assessment of manuscripts — the false appearance of relevance achieved through coercive citations can misinform the use of certain materials, the assignation of resources, and the selection of medical treatments.

Being a Sleuth in Academia

Based on my own experience, due to a conjunction of chance and curiosity, I hardly consider that being a sleuth calls for impressive capabilities. We are upheld by tenacity and driven by a sense of responsibility. In my opinion, any researcher who identifies problems when studying the literature has a moral duty to communicate as much through academic channels, such as PubPeer, with the valuable support of the Collection of Open Science Integrity Guides (box). These are two easily accessible tools to start out on the road towards becoming a sleuth.

Nevertheless, it is not easy to navigate the choppy waters of academia as a sleuth. Although knowing and working with other sleuths, whom I greatly admire, offers great satisfaction, I feel at a loss when acknowledging that we work despite very discouraging and even disagreeable situations, ranging from the passivity of some editors and publishing houses to retaliation and vilification by academic colleagues. Even so, I do not, for one moment, regret starting out on this road some two years ago.