3I
ORAL ABSTRACTS: QUANTITATIVE METHODS I
3I
Method(s): We specifically looked at the problem of assessing the risk of violence among clinical populations in a forensic psychiatric setting. We tested different models predicting violent behaviour - a simple tallying model that just sums up the evidence across all items without performing any differential weighting of the items, and a regularised regression model which can select the items which are more important than others and weigh them accordingly. We trained both models on a dataset of the HCR–20 (Historical, Clinical, and Risk Management) violence risk assessment scores and basic demographic information about the patients such as age and gender, together with a connected patient record about violent behaviour from 366 patients at a major medium-security hospital in the UK. We trained each model on 1) all available HCR-20 items and 2) only half of the HCR-20 items (10) which were selected such as to optimise each model’s out-of-sample performance.
Result(s): We found that the tallying model which only uses unit weights on the items (and has a cutoff threshold as a parameter) exhibits the same level of predictive accuracy as the more complex regression model. Moreover, we find that surprisingly, training the models on 10 predictors instead of 20 resulted in superior performance for the simple tallying model and the regularised regression model.
Conclusion(s): Our findings show that the simple tallying model performed surprisingly well in comparison to regularised regression. However, the performance of both of these models when trained on only half of the HCR-20 items, suggesting that the HCR-20 instrument can be improved by feeding it less information. This can potentially save costs and time of risk assessments, while improving diagnostic judgments, thus benefitting clinicians and patients.
Method(s): We use the example of a fixed HIV budget to be allocated across six regions. The model reflects a sub-Saharan African setting with a generalised HIV epidemic in which regions differ by HIV prevalence. Decision makers can choose to invest in one or more interventions (of late ART, male voluntary circumcision or early ART) in each region at a range of coverage levels. Costs and QALYs are predicted using a transmission model and the optimal resource allocation is identified using mathematical programming. Uncertainty is propagated through the model using Monte Carlo simulation. We assess the impact of three budgetary policies: (i) fixed budgets at the regional level, (ii) more modest planned programmes coupled with a contingency fund to preserve planned programmes, and (iii) a national budget policy which allows transfers between regions. We explore the implications of having perfect information, to represent an upper bound on the health that could be generated by data collection.
Result(s): Standard cost-effectiveness analyses overestimate the health generated under fixed regional budgets by up to 23%. This occurs as planned programmes cannot be implemented in all realisations of uncertainty. The contingency fund generates more health than regional budgets. This suggests that preserving more cost-effective interventions across realisations of uncertainty can be valuable, and that basing plans on expected costs and effects can be sub-optimal. The national budget policy outperforms these policies by allowing decision makers to maintain planned programmes via regional transfers. Perfect information outperforms the other fixed budget policies, and performs as well as soft budgets even though there is no possibility for national HIV budget over-runs.
Conclusion(s): This work shows that fixed budgets reduce health in a way that is not currently reflected in cost-effectiveness analysis. This can be mitigated via careful resource allocation decisions, budgetary policy design and information collection.
Purpose: Disease natural history models often contain parameters that are unknown or unobservable for different reasons (e.g., ethical or financial). Calibration is the process of estimating these parameters by matching model outputs to observed clinical or epidemiological data. Our objective is to compare four different calibration methods on how they perform recovering the true parameters.
Method(s): Using a known set of parameters, we used a state-transition model with four health states: Healthy, two stages of illness (S1, S2), and Dead to simulate 1,000 individuals over 30 years in a microsimulation fashion. We produced three different sets of targets: survival, disease prevalence and log-ratio between the two stages of illness. We repeated this procedure 100 times to generate multiple sets of calibration targets. We calibrated a cohort version of the model assuming three input parameters were unknown using four different approaches: 1) two goodness-of-fit (GoF) approaches based on absolute differences with equal and unequal weights, 2) a Bayesian sampling-importance-resampling (SIR) approach, and 3) a Pareto frontier approach. We considered scenarios of varying calibration target data availability with observations every 1, 2, 5 and 10 years. We compared the calibration approaches using three metrics: 1) root mean square error (RMSE) between best-fitting input sets and true parameter values, 2) the proportion of simulations in which true parameter values are contained within the bounding ellipse of best-fitting parameters (coverage), and 3) minimum quantile ellipse that contains the true parameter values.
Result(s): For the scenario with targets every 5 years (i.e., 18 calibration targets), the Bayesian approach yielded the smallest RMSE, followed by the Pareto frontier. Pareto frontier had the highest coverage, with 94% of the 95% bounding ellipse including the true parameters, followed by the GoF with unequal weights with 82%. Both GoF with equal weights and Pareto frontier had the lowest minimum coverage with 76%. The rest of the results for this scenario are shown in the table. As the number of targets increased all calibration approaches improved.
Conclusion(s): Recovering the truth depends on many system and model properties. The choice of calibration targets matter and contrary to what we expected, more targets may not necessarily be better.
We demonstrate a new method for quantifying the effects of bias adjustment on treatment decisions based on a network meta-analysis (NMA).
Method(s):
NMA combines evidence on multiple treatments from several studies to provide internally consistent treatment effect estimates and is frequently used to inform clinical guideline recommendations. Evidence from included studies is typically assessed for risk of bias using subjective tools and checklists; however these provide no information on the effects of potential bias on decisions based on the results of the NMA.
We propose a new method that provides quantitative assessment of the effects of potential bias adjustments, by deriving bias-adjustment thresholds which describe the smallest changes to the data that would result in a change of treatment decision. In other words, the treatment decision is invariant to biases within the threshold limits. Bias adjustments can be considered for individual study estimates or for overall treatment contrasts.
Bias-adjustment thresholds are derived by manipulating the Bayesian joint posterior resulting from the NMA. The amount that a given data point can change before affecting the treatment decision depends upon the influence of that data point on the joint posterior.
We also assess the effects of bias adjustment in a probabilistic cost-effectiveness analysis using inputs from the NMA. We then assess the sensitivity to bias of a treatment decision based on net benefit.
Result(s):
The threshold method was applied to a series of examples from published NICE guidelines. In most cases the treatment recommendation was robust to plausible levels of bias in all but a small proportion of contrasts or studies. In larger, well connected networks with large numbers of trials, recommendations were robust against almost any plausible bias adjustments. Sensitivity to bias adjustments for net benefit decisions resulting from cost-effectiveness analysis was also considered, showing similar results.
Conclusion(s):
Threshold analysis provides insight into the effects of bias adjustment on treatment decisions. Applying the method to treatment contrasts confers considerable flexibility, since practical applications are often based on complex models with multiple types of data input. We can have more confidence in treatment recommendations where bias-adjustment thresholds are large, and focus attention on the quality of decision-sensitive trials and contrasts, potentially reducing the need for laborious critical appraisal of all included trials.
Method(s): A sensible objective for vaccine allocation is maximizing the health benefit, defined in this paper as the number of people that escape infection. In literature this objective is often achieved by evaluating the eventual outcome of alternative allocations using numerical methods or simulation. This approach does not give a high-level explanation why certain allocations yield a higher health benefit. This is especially problematic because the resulting allocations are often inequitable and behave counter-intuitively. We propose to apply analytical methods to vaccine allocation to obtain a high-level understanding of these inequitable and seemingly counter-intuitive outcomes. Thereto, we first investigate the dependence of health benefit on the fraction of people that receive vaccination. We study the seminal SIR compartmental model to model the total health benefit as a function of the vaccination fraction that is used.
Result(s): Using implicit function analysis, we prove that the health benefits as a function of the vaccination fraction have increasing returns to scale for small vaccination fractions and decreasing returns to scale for vaccination fractions larger than a certain threshold. This implies the existence of a unique vaccination fraction that maximizes the health benefit per dose of vaccine, which we refer to as the `dose-optimal vaccination fraction'. We show that the health benefit per dose of vaccine decreases monotonically when moving away from this fraction in either direction. Surprisingly, this fraction does not coincide with the so-called critical vaccination coverage that has been advocated in literature. We show that the optimal allocation is governed by the dose-optimal vaccination fraction, as vaccinating with this fraction is the most effective way of using the available resources.
Conclusion(s): These results allow us to provide new insights into vaccine allocation to multiple non-interacting and weakly interacting populations. We explain the counter-intuitive switching type behavior of optimal allocations. We also show that allocations that maximize health benefits are rarely equitable, while equitable allocations may be significantly non-optimal.
Method(s): Using a database of published clinical trials, we conducted a quantitative benefit-risk (BR) assessment of six statins in subjects with a baseline low density lipoprotein cholesterol (LDL-C) level of 125 mg/dl (3.23 mmol/L). Treatment benefit was quantified in terms of achieved LDL-C reduction at trial completion. Risks were quantified in terms of probability of experiencing common statin side effects (myalgia (MA), transaminase (TA) elevation, and creatine kinase (CK) elevation). Using a Bayesian network meta-regression analysis, we estimated the effect of each statin on LDL-C relative to control, adjusting for LDL-C level at baseline. In a separate model, we estimated the absolute change in LDL-C achieved by control alone. We derived distributions for the achieved LDL-C levels by adding together baseline level of LDL-C, control effect, and relative effect of each statin adjusted for baseline LDL-C. Separate network meta-analyses were performed to estimate relative effects for the three side effects. Baseline estimates for the side effects were obtained by pooling placebo arms, and subsequently used to obtain absolute effect estimates. Expert opinion was elicited to develop ordinal constraints for the BR model.
Result(s): The meta-regression analysis adjusting for baseline LDL-C showed a good fit to the data, and significantly decreased heterogeneity compared to an unadjusted analysis. At a baseline LDL-C of 125 mg/dl, the scale ranges for achieved LDL-C were set at 55-125 mg/dl. Side-effect scale ranges were 0.0-0.13 for MA, 0.00-0.19 for TA and 0.00-0.06 for CK. Expert opinion indicated LDL reduction to be more important than decrease in side effect risks. With 4 forms of LDL partial value functions the best treatment was rosuvastatin (first rank probability 0.91-0.99). Uncertainty coefficients showed that further preference data was unlikely to alter first rank probabilities significantly.
Conclusion(s): When treatment benefits are quantified solely in terms of LDL-C reduction, rosuvastatin seems to have the best BR balance among the six statins in an intermediate CVD risk population. Results may be sensitive to possible confounders in the meta-regression analysis and should be examined further.