Meta-analysis of HIV-acquisition studies incomplete and unstable

Letter to the Editor

Re: Male Circumcision for the Prevention of HIV Acquisition: A Meta-Analysis

Sir,

The authors of a recent meta-analysis[1] of studies into male circumcision and HIV describe their findings as “compelling.” We disagree. They reported a remarkably high degree of inconsistency with 97% of variation across studies due to heterogeneity rather than chance (an astounding, rarely seen, level of heterogeneity). Using recently described methods[2], 28.57% of the studies would need to have been excluded to bring I² below the 50% threshold (considered high) and 32.65% excluded to bring I² below the 25% threshold considered acceptable (well above the expected 99th percentiles of 22% and 32%, respectively). Similarly, 65.51% and 65.93% of the total number of participants needed to be excluded to reach the 50% and 25% thresholds (above the 99th percentiles of 25% and 48%, respectively). Given this excessive between-study heterogeneity, Sharma et al. should have refrained from reporting summary estimates[3].

The authors half-heartedly attempted to explain the heterogeneity failing to recognise that both the risk profile and circumcision prevalence of the study population are significant factors[4], and also failing to acknowledge the sizeable percentage of iatrogenically transmitted HIV infections[5].

The authors excluded approximately half of the published studies that met their inclusion criteria[4]. Excluding studies that focused only on MSM, which have a distinctly different risk profile, we calculate the included studies as significantly more likely to report a greater treatment effect (random-effect summary odds ratio (circumcised versus intact) of (0.44, 95%CI=0.36-0.59) than the excluded studies (0.66, 95%CI=0.56-0.78) − (change in ln(OR)=0.35, 95%CI=0.07-0.65, t=2.44, p=0.016).

In assessing publication bias, the authors provided a funnel graph, declaring that the data plots “appear to be evenly distributed about the mean effect size, suggesting an absence of publication bias,” without applying any routine statistical tests. Four of six commonly used measures[6] exhibited significant publication bias.

The results of the randomised clinical trials (RCTs) have been noted as being “remarkably similar”[7]  − the probability of the results of these trials being so tightly clustered is only 0.03[8]. Can the 0% I² reported in the meta-analysis of the RCTs be interpreted as indicating no appreciable variability between the studies? This certainly arouses suspicion of prior coordination: as Ioannidis noted, “At the extreme, fraud can cause perfect replication”[9].

The large sample size RCTs allowed small numerical differences to have an exaggerated impact on p-values. The Fragility Index (FI) (number of times one patient with the relevant finding is subtracted from one group and added to another group before the results are no longer significant)[10] for the three clinical trials was 4, 5, and 6, respectively, with an FI of ≥ 8 being common, and an FI of ≤ 3 being suspect.  Early discontinuance of these fragile studies with an absolute risk reduction between 0.8% and 1.9% was an artifact of being overpowered.

Given the effectiveness of condoms[11], the lack of consistent findings on national levels[12], the methodologically flawed RCTs[13], the lack of translational research, and the impressive potential uptake and effectiveness of pre-exposure prophylaxis[14], circumcision as an intervention to prevent HIV infection should be treated with greater scepticism.

Robert S. Van Howe¹, MD, FAAP, and Gregory J. Boyle², PhD, DSc, FAPS

¹College of Medicine, Central Michigan University , Saginaw , MI , USA

²University of Melbourne, Parkville, VIC 3010, Australia

 

References

1. Sharma SC, Raison N, Khan S, Shabbir M, Dasgupta P, Ahmed K. Male circumcision for the prevention of human immunodeficiency virus (HIV) acquisition: a meta-analysis. BJU Int 2018; 121:515-26. doi:10.1111/bju.14102
2. Patsopoulos NA, Evangelou E, Ioannidis JPA. Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. Int J Epidemiol 2008; 37: 1148-57. doi:10.1093/ije/dyn065
3. Mueller M, D’Addario M, Egger M, et al. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol 2018; 18: 44.
4. Van Howe RS. Circumcision as a primary HIV preventive: extrapolating from the available data. Glob Public Health 2015; 10: 607-25.
5. Gisselquist D, Pottarat JJ, Brody S, Vachon F. Let it be sexual: how health care transmission of AIDS in Africa was ignored. Int J STD AIDS 2003; 14: 148-61.
6. Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Statist Med 2001; 20: 641-54.
7. Sansom SL, Prabhu VS, Hutchinson AB, et al. Cost-effectiveness of newborn circumcision in reducing lifetime HIV risk among U.S. males. PLoS One 2010; 5(1): e8723.
8. Van Howe RS. “Math is your friend: a consumer’s primer to understanding epidemiology.” Genital Autonomy 2014: Thirteenth International Symposium on Genital Autonomy and Children’s Rights. Boulder, Colorado. July 24, 2014.
9. Ioannidis JP. Scientific inbreeding and same-team replication: type D personality as an example. J Psychosom Res 2012; 73: 408–10.
10. Walsh M, Srinathan SK, McAuley DF, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. J Clin Epidemiol 2014; 67: 622-8.
11. de Vincenzi I. A longitudinal study of human immunodeficiency virus transmission by heterosexual partners. European Study Group on Heterosexual Transmission of HIV. N Engl J Med 1994; 331: 341-6.
12. Garenne M. Long-term population effect of male circumcision in generalised HIV epidemics in sub-Saharan Africa. Afr J AIDS Res 2008; 7: 1-8.
13. Boyle GJ, Hill G. Sub-Saharan African randomised clinical trials into male circumcision and HIV transmission: methodological, ethical and legal concerns. J Law Med 2011; 19: 316-34.
14. Baeten JM, Donnell D, Ndase P, et al. Antiretroviral prophylaxis for HIV prevention in heterosexual men and women. N Engl J Med 2012; 367(5): 399-410.