A COMPARISON OF ALTERNATIVE STATISTICAL METHODS FOR COST-EFFECTIVENESS ANALYSES THAT USE DATA FROM CLUSTER RANDOMIZED TRIALS

Purpose: Cost-effectiveness analyses (CEA) may be undertaken alongside cluster randomized trials (CRTs) where the unit of randomization is the cluster (e.g. hospital) not the patient. This paper compares for the first time statistical methods for CEA that use data from CRTs: multilevel models (MLMs), Generalised Estimating Equations (GEEs), and a 2-stage non-parametric Bootstrap which re-samples clusters and then cases within clusters.

Method: Bivariate GEEs and the Bootstrap are relatively simple to implement compared to bivariate MLMs but rely on asymptotic assumptions, which with few clusters, may not be satisfied. We initially compare the methods with data from a large (1732 cases, 70 primary care clinics) CRT evaluating an intervention in primary care for reducing post-natal depression. We then undertake an extensive simulation study that compares the relative performance (bias, mean squared error and confidence interval coverage) of each method for estimating mean incremental net benefits (INB). Methods were initially tested under relatively ‘ideal circumstances’: normally distributed costs and QALYs, CRT with many (n=100) clusters and balanced cluster size (50 cases in each cluster). Then, we test the methods in scenarios with more realistic characteristics, based on our systematic literature review of 62 published studies: a)    Few (<10) balanced clusters b)    Many (40) imbalanced clusters c)    Few (<10) imbalanced clusters d)    As above but assuming data are gamma distributed

Result: The case study found similar mean INBs (λ=£20,000 per QALY) across methods but the 95% CI were much wider for the Bootstrap method (mean £135, 95% CI from -£60 to £549) versus the MLM (£58, -17 to 138), and GEE (£98, -24 to 220). The simulation study showed that under the ideal scenario all three methods performed relatively well. Under more realistic scenarios, the performance of all the methods was seriously affected in terms of MSE and coverage but not bias. When there were few (e.g. 6) balanced clusters, many imbalanced clusters, or few imbalanced clusters, the Bootstrap generally performed worst. For example, coverage levels were too conservative, and MSE was 25% higher than for the other methods. When cost data were simulated from a gamma distribution, the MLMs and GEEs continued to outperform the Bootstrap

Conclusion:  The 2-stage Bootstrap appears to perform worse than MLMs and GEEs across many circumstances faced by CEA that use CRTs.