Candidate for the Lee B. Lusted Student Prize Competition
Method: From the Cochrane Library (Issue 4, 2012) we randomly selected 50 reviews that searched the Pubmed/MEDLINE and EMBASE databases and meta-analyzed at least 10 studies. For each eligible citation in each review, we checked whether it was retrievable in PubMed and EMBASE. For the first-listed meta-analysis of 10 or more studies in each review we examined whether excluding studies not found in PubMed or in EMBASE changes summary estimates.
Result: The 50 Cochrane reviews included a median of 39 papers (25th-75thpercentile: 19-75). The median numbers of PubMed-, EMBASE-, and non-PubMed/non-EMBASE-indexed papers were 29 (17-53), 33 (17-57), and 4 (1-9), respectively. Searching EMBASE in addition to PubMed contributed a median of 1 (0-3) more papers. The selected meta-analyses included a median of 14 studies (12-19), of which a median of 12 (10-16) were indexed in PubMed and 13 (11-17) in EMBASE. A median of 0 (0-1) studies per meta-analysis were identified through EMBASE but not PubMed. Relying only on PubMed would change the statistical significance in 1 out of 50 meta-analyses compared to using all studies (P=0.024 vs. P=0.068, respectively). In meta-regressions PubMed-indexed studies had a significantly different effect size compared to non-PubMed-indexed studies in only 2 datasets.
Conclusion: The gains from searching sources beyond PubMed, and from searching EMBASE in particular, for systematic reviews, may be more modest than commonly believed.