Tag Archive for: Jae Hung Jung

You are here: / Jae Hung Jung

Posts

Reaching for the stars – rating the quality of systematic reviews with the Assessment of Multiple Systematic Reviews (AMSTAR) 2

/by

The number of published systematic reviews and meta‐analyses in the urological literature has dramatically increased in recent years [1]. This is good news given their importance in guiding clinical decision‐making, guideline development and health policy. However, many of these studies are of low quality, raising concerns about the trustworthiness of their results. As with other research studies, it is therefore important for readers to have a framework for determining the quality of a given systematic review. Therefore, in 2017 BJU International launched a scoring system for systematic reviews that provides readers with a summary assessment as to whether established methodological safeguards against bias for systematic reviews have been met [2]. This is based on the Assessment of Multiple Systematic Reviews (AMSTAR), a validated instrument that assesses methodological quality on an 11‐point scale (0–11), with higher scores reflecting greater methodological rigor and all criteria being given the same relative weight [3].

Recently, an updated version of this instrument has become available, offering a better assessment of systematic reviews [4]. The revised instrument (AMSTAR 2) includes 10 of the original domains; it has 16 items in total (compared with 11 in the original), simpler response categories to the original AMSTAR, and provides an overall rating that is largely based on seven critical domains that should all be met. These relate to: (i) documentation of an a priori registered protocol in Prospective Register of Systematic Reviews (PROSPERO) or through Cochrane, (ii) a comprehensive literature search, (iii) explicit justification for excluding studies, (iv) a risk of bias assessment of included studies, (v) appropriate use of meta‐analytical methods, (vi) consideration of risk of bias when interpreting the results of the review, and (vii) assessment of presence and likely impact of publication bias. Other, non‐critical domains include a clear description of the study question in Population, Intervention, Comparison, Outcome (PICO) format, study selection and data extraction in duplicate, and identification of sources of funding of the studies included in the review and the review itself. This results in a four‐tiered rating (high, moderate, low, and critically low) that reflects the confidence that a reader may place in the results. Notably, a high‐quality rating requires no critical weakness and allows for only one non‐critical weakness. More than one non‐critical weakness drops the rating down to moderate, and just one critical weakness (such as lack of an a priori protocol) drops the rating down to low. Any review that has more than one critical weakness will be rated as critically low.

BJU International editors will routinely apply this AMSTAR 2‐based scoring system to screen for methodological quality in order to raise the awareness of this issue and promote reviews of higher quality (Fig. 1)[1]. Needless to say, BJU International is not the place for systematic reviews of sub‐optimal methodological quality in which the readers cannot place their trust. Meanwhile, we also fully understand that methodological rigor is not everything but has to be paired with clinical relevance and newsworthiness. Much has been written about the dramatic redundancy of systematic reviews on the same topic; in certain areas of medicine, the number of systematic reviews exceeds that of eligible studies that these reviews included [5]. Therefore, when systematic reviews already exist, there needs to be a clear rationale for any ‘encore’ performance. BJU International also encourages the development of systematic reviews by author teams that are financially unconflicted and have thoughtfully managed any intellectual conflict of interest.

Figure 1: New BJUI rating system of systematic reviews based on AMSTAR 2. The number of coloured stars in the inner and outer layers of the system represents completeness of an individual critical domain and overall confidence rating of the systematic review, respectively. The number in the middle of the system refers to the summary AMSTAR 2 score based on the overall confidence rating of the systematic review (high: 4, moderate: 3, low: 2, critically low: 1).

Through this initiative, BJU International not only intends to become the premier journal for high‐quality systematic reviews as they relate to urology, but also to move the field forward, reducing redundancy and waste. As we embrace the higher standards of AMSTAR 2, we present the first review to be scored using this method in this issue [6] and we encourage all systematic review authors to accept this challenge and reach with us for the stars.

References

  1. Han JL, Gandhi S, Bockoven CG, Narayan VM, Dahm P. The landscape of systematic reviews in urology (1998 to 2015): an assessment of methodological quality. BJU Int 2017; 119: 638–49
  2. Dahm P. Raising the bar for systematic reviews with Assessment of Multiple Systematic Reviews (AMSTAR). BJU Int 2017; 119: 193
  3. Shea BJ, Grimshaw JM, Wells GA et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007; 7: 10

 

About the authors:

Dr Philipp Dahm is Professor of Urology and Vice Chair of Veterans Affairs at the University of Minnesota. He also serves as Director of Research and Education for Surgical Services at the Minneapolis Veterans Administration Medical Center (@EBMUrology).

 

Dr Jae Hung Jung is from the Department of Urology, Wonju College of Medicine, Yonsei University, Korea.

 

 

 

The challenge with systematic reviews of non-randomised studies in urology

/1 Comment/by

In this issue, BJU International has made the conscious decision to publish a systematic review (SR) and meta-analysis (MA) by Guo et al. [1]to inform the question of whether patients undergoing nephrouretectomy for upper tract urothelial carcinoma are at increased risk of worse oncological outcomes. This question was also the topic of a similar review by Marchioni et al. [2] published earlier this year in this journal. Both studies were submitted around the same time and underwent independent, parallel peer review that resulted in different editorial decisions. Given their similarity in methodological quality they both deserved similar consideration for publication, which the journal is hereby honouring.

At the same time, this provides the unique opportunity to reflect on methodological developments in the field and BJU International‘s efforts to raise the bar of the methodological quality of SRs, which include the provision of an Assessment of Multiple Systematic Reviews (AMSTAR) rating [3]. AMSTAR is a validated tool to assess the components of a SR on an 11-point scale (0–11), with higher scores reflecting higher methodological rigor. An updated version of this tool has recently been provided, which offers greater clarity in interpretation [4]. Another related instrument that has become available is the Risk of Bias in Systematic Reviews (ROBIS), which assesses the study limitations in SRs (i.e., the relevance of the review, concerns with the review process, and potential bias introduced during the review) [5]. Meanwhile, while it would be premature to claim success, it is our impression that BJU International’s initiative to provide AMSTAR ratings is making a valuable contribution in raising awareness for such methodological issues and improving the transparency of published reviews.

As BJU International takes a lead in promoting high-quality SRs in urology, the journal has seen a considerable increase in the number of submissions, including SRs of non-randomised studies (NRS). Whilst much of what we practice on a day-to-day basis is based on evidence from NRS, studies of those designs have infrequently been included in the Cochrane Library, which has pioneered much of the underlying methodology. This is for a few reasons: First, the ‘garbage in–garbage out’ phenomenon; if the underlying individual studies only provide very low-quality evidence, combining these studies will rarely enhance the confidence we place in their results. Second, the need for methodological advances in the assessment and analyses of NRS. Third, when high-quality evidence from randomised controlled trials (RCTs) is available, it may be inefficient to review the NRS literature.

However, progress is being made on the methodology front. Members of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group are credited with having developed an approach for rating the quality or certainty of evidence from randomised and NRS to inform decision making [6]. While a body of evidence from RCTs, which starts as high-quality evidence, may be downgraded for study limitations, a body of evidence from NRS, which starts as low-quality evidence, may be upgraded for one of three reasons, most commonly for large magnitude of effect [7]. The underlying assumption is that, whilst we have to assume that bias is likely to be present in these studies, it is unlikely to explain the entire observed effect.

It nevertheless remains critical to assess the risk of bias of NRS. While the Newcastle-Ottawa scale (as used in both of these SRs) is a widely used instrument to evaluate risk of bias in NRS, it has critical limitations that the recently developed Risk Of Bias In Non-randomised Studies – of Interventions (ROBINS-I) seeks to overcome [8]. ROBINS-I evaluates NRS by using a standardised comparison to an RCT (i.e. target trial) [9]. In this way, ROBINS-I captures the bias inherent to studies without proper randomisation or allocation concealment, namely the lack of a balance of known and unknown confounders and selection of participants. ROBINS-I allows users to fundamentally start all studies at the same quality level, providing the transparency requested by some SR authors conducting SRs of NRS.

While ureterorenoscopy before nephroureterectomy may indeed increase the risk of intravesical recurrence as the authors suggest, additional exploration would be needed to make a statement about the causality of the relationship. Guo et al. [1] conducted sensitivity analyses to describe the potential for bias introduced by confounders of previous bladder tumour history and bladder-cuff management, thereby increasing our confidence that the observed effect may be closer to the truth. It seems equally important to note that Guo et al. found no increased risk in cancer-specific, recurrence-free or overall survival, which are other outcomes of potentially greater patient importance.

Understanding the inherent limitations of NRS, and placing their findings into appropriate clinical context are critical to the conduct of SRs. Moving forward, BJU International will continue to seek out the highest quality reviews that make use of the best, up-to-date methodology. We hope that these efforts will both serve as a beacon for the research community, but more importantly, result in improved evidence-based care for our patients.

Philipp Dahm*, Jae Hung Jung† and Rebecca L. Morgan‡
*Department of Urology, Minneapolis VA Medical Center, University of Minnesota, Minneapolis, MN, USA, †Department of Urology, Yonsei University Wonju College of Medicine,
Wonju, Korea and ‡Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada

 

References

 

1 Guo RQ, Hong P, Xiong GY et al. Impact of ureteroscopy before radical nephroureterectomy for upper tract urothelial carcinomas on oncological outcomes: a meta-analysis. BJU Int 2018; 121: 184-93

 

2 Marchioni M, Primiceri G, Cindolo L et al. Impact of diagnostic ureteroscopy on intravesical recurrence in patients undergoing radical nephroureterectomy for upper tract urothelial cancer: a systematic review and meta-analysis. BJU Int 2017; 120: 313–9

 

3 Dahm P. Raising the bar for systematic reviews with Assessment of Multiple Systematic Reviews (AMSTAR). BJU Int 2017; 119: 193

 

4 Shea BJ, Reeves BC, Wells G et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017; 358: j4008

 

5 Whiting P, Savovic J, Higgins JP et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016; 69: 225–34

 

6 Guyatt GH, Oxman AD, Vist GE et al. GRADE: what is “quality of evidence” and why is it important to clinicians? BMJ 2008; 336: 995–8

 

7 Guyatt GH, Oxman AD, Sultan S et al. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol 2011; 64: 1311–6

 

8 Deeks JJ, Dinnes J, D’Amico R et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003; 7: iii–x, 1–173

 

9 Sterne JA, Hernan MA, Reeves BC et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016; 355: i4919

 

© 2024 BJU International. All Rights Reserved.