Purpose : To review measures and
graphical displays to assess the incremental predictive value of markers over
standard, readily available
characteristics.
Methods : We evaluate
the incremental value of adding HDL to a Cox regression model in 3264 subjects from
the Framingham study to predict 10-year risk of coronary heart disease (n=183 events, Pencina Stat Med 2008). Statin
therapy is considered indicated for subjects with higher than
20% 10-year risks.
A traditional measure for
incremental values is the improvement in the area under the
ROC curve (AUC). New measures include
the net reclassification improvement
(NRI) and decision-analytic
measures, such as the net
benefit (NB).
Results : The AUC difference (ΔAUC) between a
model with and without HDL
was small in numerical value
(0.012 for adding HDL with continuous risk; 0.029 with 20% threshold). Using a 20% threshold to classifiy
subjects as high risk, the 2 NRI components are the
net percentages of correctly reclassified
patients with events
(11/183=6.0%) and without events (-5/3081=-0.2%). Their sum is the NRI (0.058, which is higher in numerical value than ΔAUC, 0.029).
The NB is the net fraction of true-positive (TP) classifications penalized for false-positive (FP) classifications: NB= (TP - w FP)/N, with w defined by the odds of harms:benefits, or odds(threshold). A threshold of 20% implies w=0.2/(1-0.2)=0.25. ΔNB can hence be calculated as (11 – 0.25*5)/3264=0.30%.
For better understanding
of the NRI and NB, we propose
a "Net Reclassification Risk" graph.
This simple graph allows us
to focus on the number of patients and event rates of the 2 reclassified groups: those reclassified
from high to low risk (H/L,
n=29, 10% event rate) and those reclassified from low to high risk (L/H, n=45,
31% event rate). We note that 45*0.31 – 29*0.10 = 11 events can be extra identified
(ΔTP=11, NRIevents 11/183, 6%). The negative side is that 45*(1-0.31)
– 29*(1-0.10) = 5 extra overtreatments are expected (ΔFP=-5, NRInonevents
-5/3081, -0.2%). The burden of overtreatment
is explicitly weighted by 0.25 in the NB calculation, leading to the 0.30% estimate for NB
((11-1.25)/3264=0.30%).
Conclusions : More modern and decision-analytic reclassiifcation measures can be better
understood to assess the incremental predictive value of a marker by a simple graph
for the Net Reclassification
Risk ('NRR graph').
