GETTING THE BEST OF BOTH WORLDS: AN APPROACH FOR MAXIMISING INTERNAL AND EXTERNAL VALIDITY IN COST-EFECTIVENESS STUDIES

Purpose: Cost-effectiveness analyses (CEA) may use RCTs to maximise internal validity. However, when RCTs include patients and centres atypical of those in routine clinical practice, CEA results may be subject to sample selection bias. To reduce this bias, observational data can be used to reweight the trial-based estimates. We present an approach to assess the assumptions behind any reweighting strategy, illustrated with a case study of high policy-relevance.

Method: We decompose sample selection bias into observable or unobservable differences between the RCT and the setting of interest. We consider alternative ways of reweighting the RCT estimates, to the population’s characteristics. The first estimation strategy, reweights according to Inverse Probability of Treatment Weighting (IPW), where ‘treatment’ is inclusion in the RCT. The second strategy uses maximum entropy (MaxEnt) weighting along with matching. Either approach makes the identifying assumption that selection into the RCT is conditional on observable characteristics. We consider this critical underlying assumption with novel placebo tests. These test the non-equivalence of reweighted outcomes following treatment in the RCT, versus outcomes after treatment in the population. Passing these tests implies that the identifying assumption holds, and there is sufficient power to detect outcome differences across settings. We consider this approach in a UK CEA of Pulmonary Artery Catherization (PAC) using an RCT (n=1,014), and observational data on PAC use in routine practice (n=1,000).  Across both settings, 40 baseline covariates were identically recorded. Differences across settings were reported, for example in the proportion admitted to non-teaching hospitals (RCT: 80%; population: 60%). We used IPW and MaxEnt to reweight the RCT estimates. We report cost-effectiveness overall, and for subgroups defined a priori.

Result: The overall incremental net benefit (INB) of PAC from the RCT was -£7,900 (95% CI from -£18,500 to £2,600), the corresponding estimates reweighted for the general population were, -£10,000 (-18,500 to -£2000) [IPW] and £1,500 (-£6,700 to £9,900) [MaxEnt]. For non-teaching hospitals, the INBs were £900 (-£12,100 to £14,000) [RCT], £200 (-£9,900 to £10,300) [IPW] and £18,800 (£8,400 to £29,200) [MaxEnt]. IPW failed placebo tests both overall and for the non-teaching hospital subgroup, whereas MaxEnt passed the corresponding tests.

Conclusion: This approach can help maximise the external validity of RCT-based CEA. The placebo tests presented are useful for choosing amongst competing weighting strategies.