February Editorial: Raising the bar for systematic reviews with Assessment of Multiple Systematic Reviews (AMSTAR)

The BJUI has a longstanding track record in promoting the dissemination of high-quality unbiased evidence and helping their readership to understand why the principles of evidence-based medicine matter. This devotion is witnessed by the work that goes into every issue of the journal, as well as past initiatives such as providing a level of evidence rating for clinical research articles or publishing educational articles such as the ‘Evidence-Based Urology in Practice’ series [1, 2].

Major foci for clinically oriented specialty journals are systematic reviews and meta-analyses. Systematic reviews have a preeminent role in guiding the practice of evidence medicine by addressing focused clinical questions in a systematic, transparent and reproducible manner. Defining criteria of a high-quality systematic review include: an a priori registered protocol, a comprehensive search of multiple sources including unpublished studies (to avoid publication bias), an assessment of the quality of evidence that goes beyond study design alone, and a thoughtful interpretation of the findings. Systematic reviews inform clinicians and patients at the point of care, form the foundation of evidence-based clinical practice guidelines, and help shape health policy [3]. They also find frequent citation and can raise a journal’s impact factor. There is therefore more than one good reason for journals to care about the quality of systematic reviews.

Meanwhile, a study in this issue of the BJUI [4] shows that the methodological quality of systematic reviews published in the urological literature is modest, varies substantially, and has failed to improve over time. This contrasts to randomised controlled trials’ reporting quality that appears to have improved substantially over time, probably due to increased awareness among clinical researchers, urology readers and journal reviewers [4, 5]. The study [4] used the Assessment of Multiple Systematic Reviews (AMSTAR), a validated 11-item instrument, to measure the methodological quality of systematic reviews with higher scores reflecting better quality.

The authors [4] surveyed four major urological journals and compared the periods 2013–2015 to 2009–2012 and 1998–2008. Despite a dramatic increase in the number of systematic reviews published each year, methodological quality has stagnated with mean AMSTAR scores ± standard deviations of 4.8 ± 2.4 (2013–2015; n = 125), 5.4 ± 2.3 (2009–2012; n = 113) and 4.8 ± 2.0 (1998–2008; n = 57). The average systematic review therefore has deficits in over half the 11 AMSTAR criteria and is of only modest quality thereby undermining our confidence in their results. Although the mean AMSTAR score of 5.6 ± 2.9 for 25 systematic reviews published in the BJUI in 2013–2015 compared favourably to similar studies in other leading urology journals, the difference was not statistically significant.

What are we going to do about it? Inspired by these findings, the BJUI is launching a new initiative to raise awareness for the issue of methodological quality of systematic reviews among its readership and raise the bars for its contributors. Future systematic review authors will be asked to submit an AMSTAR-based checklist to provide enhanced transparency about its methods that will be reviewed as part of the editorial review process. These include documentation of an a priori written protocol and ideally, registration of the systematic review through the Cochrane Collaboration or the Prospective Register of Systematic Reviews (PROSPERO). Such a protocol should outline all important steps of the review process including the definition of outcomes, study inclusion and exclusion criteria, details about the literature search, study selection and data abstraction process, analytical approach including planned sensitivity and subgroup analyses. Authors should also rate the quality of evidence looking beyond study limitation alone by using an approach such as the Grading of Recommendations Assessment, Development, and Evaluation (GRADE), which recognises such additional domains such as imprecision, inconsistency, indirectness and publication bias [6]. Critical steps of the systematic review process should be completed in duplicate to guard against random and systematic error and authors should provide readers with the information about who funded the studies included in the review, as well as their own potential conflicts of interests. To guard against publication bias, systematic review authors should also search for ongoing trials and unpublished studies through registries and abstract proceedings.

It is understood that the methodological handiwork that goes into the planning, execution and reporting of a systematic review do not assure clinical relevance or newsworthiness, nor does it address any issues surrounding the limited quality of studies that the review may be summarising. However, it is nevertheless a sine quae no to assure readers that they can be confident of the results. The new BJUI initiative will raise awareness for the issue of systematic review quality by providing a summary AMSTAR score to accompany each article. We hope that with this initiative we will provide a beacon for other specialty journals to follow, with the goal of raising the bar for all published systematic reviews and ultimately leading to improved patient care.