4K-1 EXPLORING THE USE OF DECISION THRESHOLDS FOR RANKING TREATMENTS IN NETWORK META-ANALYSIS

Tuesday, October 25, 2016: 3:30 PM
Bayshore Ballroom Salon E, Lobby Level (Westin Bayshore Vancouver)

Romina Brignardello, PhD1, Alejandro Jadad, MD, DPhil2, Bradley Johnston, PhD1, George Tomlinson, PhD1 and Nicholas Mitsakakis, MSc PhD3, (1)Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada, (2)Institute for Global Health Equity and Innovation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada, (3)Toronto Health Economics and Technology Assessment (THETA) Collaborative, Toronto, ON, Canada

Purpose: The ability of Markov Chain Monte Carlo (MCMC) Bayesian network meta-analysis (NMA) to rank treatments is one of its most appealing features; however, rankings of treatments may be misleading if they are not considered together with estimates of their relative effectiveness. The purpose of this work was to explore the robustness of the rank probabilities obtained from Bayesian NMA by calculating them under increasingly stringent thresholds for the relative effect that defines a treatment effect difference.

Method: We modified the usual rankings procedure for Bayesian NMA to allow that two MCMC samples of treatment effects had to differ by a non-zero amount before one effect would be considered better than the other.  On the odds ratios scale, we examined thresholds for the relative effect from 0.6 (a large difference) up to 1 (any difference). We applied this revised rankings procedure to all published systematic reviews using NMA from the field of cardiovascular medicine that had trial-level binary data available.  We reran all the NMAs and in each one, for the two treatments identified as being the best, examined the effect on the rank probabilities of using increasingly stringent decision thresholds.

Result: We included 14 systematic reviews, having a median of 20 randomized trials and 9 treatments. The best treatments had rank probabilities that ranged from 38% to 85.3%. The effect of increasing the stringency of the decision thresholds on the probability of a treatment being best varied across reviews, with the probability of being best changing less than 20% in the most robust settings, but decreasing to almost 0% in the least robust. 

Conclusion:  Rank probabilities can be fragile to changes in the decision threshold used to claim that one treatment is more effective than another.  Our revised procedure that includes these thresholds in the calculation of rankings may aid their interpretation and use in clinical practice.

Figure: Example of rapidly decreasing probability of a treatment being best when clinically important thresholds are considered.

See more of: ORAL ABSTRACTS: OPTIMIZING DECISION MAKING

See more of: 38th Annual North American Meeting of the Society for Medical Decision Making