THE COST OF BEING WRONG: THE IMPACT OF PREDICTION UNCERTAINTY ON THE COST-EFFECTIVENESS OF RISK-STRATIFIED STRATEGIES

Purpose: We demonstrate an approach to assess the impact of uncertainty in risk predictions on health-economic outcomes in risk-stratified prevention strategies, illustrated for preventive statin treatment based on 10-year coronary heart disease (CHD) risk predicted by the Framingham risk score (FRS).

Method(s): A Markov decision-analytic model was used to simulate cohorts with preventive statin treatment. We fitted the FRS to men and women from the Atherosclerosis Risk in Communities (ARIC) cohort. Using the ARIC risk distributions hypothetical cohorts of men and women aged 50–59 years followed for 30 years. Individuals were preventively treated if their predicted CHD risk exceeded treatment threshold T.  While lowering the threshold T from 20% to 0% (1% decrements), strategies including gradually more and more treated individuals were evaluated. Assessing quality-adjusted life-years (QALYs) and costs at each step, the Net Health Benefit (NHB) (willingness-to-pay of $50,000/QALY) of treating an individual with a certain predicted risk was calculated.

Subsequently, the FRS was refitted to 1,000 bootstrap samples of men and women from the ARIC cohort of varying sizes (N=5,000;N=2,500;N=1,000), while rejecting models not achieving en acceptable level of performance (AUC>0.6). Using these refitted models, we calculated 1,000 alternative individual risk predictions. We then assessed whether a different treatment decision would have been made when applying the alternative risk predictions. Finally, we matched each alternative risk prediction to the corresponding NHB to estimate the impact of the uncertainty a predicted risk on the NHB.

Result(s): Preliminary results indicate that prediction uncertainty resulted in probabilities of incorrect treatment decisions of up to 0.34 and 0.47 (N=5,000), 0.40 and 0.49 (N=2,500), and 0.47 and 0.55 (N=1,000) for predicted risks surrounding T=5% and T=20%, respectively (Figure). The risk-based NHBs ranged from K for a predicted risk p=L% to K for p=L% in men and from K at p=L% to K for p=L% in women.

Conclusion(s): While uncertainty in risk predictions may lead to incorrect treatment decisions, associated impact on long-term health-economic outcomes is often unknown. Assessing this impact can guide studies aiming to improve prediction models by focusing on improving risk prediction in individuals for which improvement may actually improve health-economic outcomes.