Filtering failure: the impact of automated indexing in Medline on retrieval of human studies for knowledge synthesis

Abstract

Objective: Use of the search filter ‘exp animals/ not humans.sh’ is a well-established method in evidence synthesis to exclude non-human studies. However, the shift to automated indexing of Medline records has raised concerns about the use of subject-heading-based search techniques. We sought to determine how often this string inappropriately excludes human studies among automated as compared to manually indexed records in Ovid Medline.

Methods: We searched Ovid Medline for studies published in 2021 and 2022 using the Cochrane Highly Sensitive Search Strategy for randomized trials. We identified all results excluded by the non-human-studies filter. Records were divided into sets based on indexing method: automated, curated, or manual. Each set was screened to identify human studies.

Results: Human studies were incorrectly excluded in all three conditions, but automated indexing inappropriately excluded human studies at nearly double the rate as manual indexing. In looking specifically at human clinical randomized controlled trials (RCTs), the rate of inappropriate exclusion of automated-indexing records was seven times that of manually-indexed records.

Conclusions: Given our findings, searchers are advised to carefully review the effect of the ‘exp animals/ not humans.sh’ search filter on their search results, pending improvements to the automated indexing process.