Excessively reproduced research: Detection of plagiarism in PubMed abstracts

Most of the plagiarism detection systems employ some kind of fuzzy matching algorithm. For example, Crossref, from Elsevier, uses a system called ``iThenticate'' to identify duplicates. It's proprietary, so hard to get meaningful numbers.

My goal in this project was very simple: find all the sentences in PubMed abstracts that appear more than once? I required an exact, character-for-character match. I wanted to do this for many abstracts (millions).

The data are the approximately last 8 years of PubMed XML files (up to 2018). The way PubMed works is that they release new batches in XML format. But they're not just the latest publications. They also include updates to prior publications, and each entry in the XML file contains a version number. Therefore some of the entries in a recent file could be updates of earlier publications. It's a little bit difficult to say exactly what the date range I used, but roughly speaking, it's the last eight years.

Here's a cartoon version of my algorithm. I make a dictionary (hash table) that points from each sentence to a list of abstracts containing that sentence. To carve up each abstract into sentences, I use a rather complex regular expression. It doesn't work perfectly, but that doesn't matter because it's deterministic. The same abstract going through each time will get parsed the same way.

Basically I'm making an inverted index from sentences to all the abstracts that contain those sentences.

index = dict()
for each abstract
   for each sentence
      index[sentence] += abstract

For the second step, find all sentences that occur in more than one abstract.

for each (sentence, abstracts) in index
    if length(abstracts) > 1
        record sentence as possible suspect

The actual code is much longer and more complex. To get this to run on my laptop, because the XML files are rather large, the code makes multiple passes over the data to reduce the storage space needed. (This also speeds the processing.) All of these details have been omitted here. If you had enough RAM and storage for a database, you could implement something very close to my cartoon algorithm.

Although requiring an exact character-for-character match sets a very high threshold of detection, there are still false positives. I introduced a set of heuristics to remove some kinds of false positives:

• Remove sentence if occurs in more than 3 abstracts
• Remove sentence if too short
• Remove abstract if it is a ``Reports an error in''or ``Reports the retraction of''
• Remove abstract if it has fewer than 3 sentences
• Remove sentence if abstract appears in journal that prints abstracts from other journals as part of a commentary
• Remove pairs of abstracts sharing a sentence if there are any authors in common

I will try to motivate at least some of these heuristics. If a sentence occurs in a lot of abstracts, it's not stolen, it's boilerplate text, probably something as simple as a copyright notice. If you follow the PubMed rules, copyright notices aren't supposed to appear in the abstract, but some publishers don't follow these rules.

Really short sentences, just by virtue of their length, have a relatively high probability of recurring if you process many abstracts; these sentences are just not that interesting.

Really short abstracts are also not interesting; there's just not that much information within.

Some abstracts appear in article entries that are pointers to other entries in these XML files. So the abstract is an partial or exact duplicate. One way this happens is when entries are linked by ``reported error in'' or ``report the retraction of''.

Another issue is self-plagiarism. There are people who sometimes publish the same thing more than once. In certain circumstances, that's completely legitimate. In other cases, it's highly frowned upon, and I didn't want to get into trying to adjudicate this issue. So I decided to adopt a liberal policy that if there's a non-null intersection between the author lists of two suspect abstracts, then that's acceptable. I'm only retaining the abstract pair when there's complete lack of overlap in the authors.

As noted earlier, sometimes things like copyright information appear widely, and sometimes the second article is a commentary, and shares the same abstract.

My shared-author heuristic doesn't work in some circumstances, such as for the Spanish naming custom of having two last names. Depending upon which last name is listed as the last name of the author my heuristic might fail.

In some countries, sometimes one's last name is written first; the journal may or may not represent that correctly. If they don't do the right thing then my heuristic fails.

There are certainly entirely acceptable circumstances in which, for example, clinical guidelines are published simultaneously in multiple journals.

And it seems like at least within the community of people who do meta analyses, it may be legitimate to start with a previous meta analysis, particularly, if it was conducted by your team and then updated five or 10 years later. If so, then some of that text might be identical.

What are the real true positives? It looks very likely that they are text used without attribution; in some cases this applies to entire articles.

Here are some examples. These were handpicked by me to illustrate some points. Automatically picking high-likelihood candidates is not a problem I have solved.

What I produce is a large HTML file; this is so that it can be easily viewed in a browser, which allows reformatting of size and width, and also supports hypertext links.

What we see here is a pair of articles. Each article's entry starts with the light green background. That line shows the PubMed ID, the list of authors, the journal name, a link to the PubMed entry, and the DOI link for the actual article. The second article in the pair is below and appears in the same format.

The colors indicate sharing. This yellow sentence starting ``Posterior pedicle screw fixation ….'' is exactly the same sentence in both abstracts. You can see there are some other short shared sentences as well.

The light blue marks a sentence that is shared with a different abstract that is not a member of this pair.

My hypothesis for what happened here is that the second article (from 2017) used without attribution text from the first article (from 2013). The second article does not cite the first one. Both are meta analyses, but with slightly different foci. I suspect the second author just borrowed some text because he was lazy. Or he maybe was in a rush.

Here we have another pair of articles, with different authors, different journals, and the same year. A couple sentences are exactly the same, but it's not entirely clear when you read the rest of the abstract what's going on. If you dive into the article text, you find that it is nearly entirely identical.

Note that it is not clear in this instance who borrowed from whom. Check out these articles and decide for yourself: https://pubmed.ncbi.nlm.nih.gov/28128425/ and https://pubmed.ncbi.nlm.nih.gov/28386191/

They did change the name of the hospitals where the research was conducted. They also have different conclusions. My hypothesis is that the ``borrower'' maybe ran up against a word limit and had to shorten their conclusion. I don't really know. Note that they have different acknowledgments and list different funding sources.

In this third example abstract, there's only one sentence that's shared, although some of the couple of sentences look similar.

This one is very suspicious. The articles were published two years apart. In the article text, there are many borrowed sentences. The figure below shows one such example which I've marked by hand; the sentences vary by one word, and the references 20–23 are the exact same references as 39–42 in the second paper. Note that Regulski's paper does not cite Tominaga's earlier paper. With some investigation, I found out that Regulski is on the advisory board of the journal that published this paper.

There's various personality factors, codes and mores, the dynamics of groups, and how they respond to such things. Laws and regulations, and culture also matter, and incentives certainly have effects on behavior, ethical and otherwise.

Just recently, China banned cash rewards for publishing. Obviously China is a very large country, and there are many scientists in China, who may have been given improper incentives to publish and perhaps publish things that maybe weren't even theirs. That China has now at least removed one of these incentives is a good sign.

I did not use a probability model to determine how likely a sentence is. This depends a lot on the sentence length and the content of sentence. But I did not try to unleash the awesome power on machine learning. I was doing something very simple.

If you had reasonable output from a classification system, followed by a stage for human assessment, you could start doing some interesting things, looking for patterns by author, by institutions, by country, by journals, by journal types, or by impact factors. But without accurate classification this is hard to do, but we had that we could say some interesting things about the current state and maybe help shape policy.

The fact that somebody is on the editorial board and is also feels perfectly comfortable adapting text from other articles which he doesn't even bother to cite, and publishing it in a journal he has a strong affiliation with—it might be nice to see where all these people lie within a social network.

If you have access to a server and you have all the PubMed abstracts on that server, you could do something as straightforward as that very cartoon version of the algorithm.

Ideally, this would be part of a larger workflow, where the new XML updates would come in, you'd register them to database, reconcile different versions of these things, run some code and then have a team of people who reviewed positives by hand and then the fun begins. You then email the journal editors of the implicated articles and say, ``Hey, I look at which I found. What do you think?''. If they don't get back to you with three or six months, you post everything on a website of miscreants.

I think NLM should also consider doing something like with all submissions to PubMed. Or Google—they already have an inverted index of most of the Web.

Finally, there's a very interesting blog, called Retraction Watch, that I highly recommend, although it's a bit discouraging. They track retractions of scientific publications. Some of them are real ethical miscues—they are just people who have stolen content or made stuff up. In other cases, it is a bit harder to tell. And if what I propose became a real thing, I think that Retraction Watch would be very interested in hearing about the results.