Purpose: Recently, the Centers for Medicare and Medicaid services (CMS) considered covering computed tomography colonography (CTC) to screen for colorectal cancer (CRC) in average-risk individuals. A technology assessment based on 3 established CRC models was presented as part of the national coverage determination process to the Medicare Evidence Development & Coverage Advisory Committee (MEDCAC), an independent advisory group charged with making recommendations to CMS. We sought to determine whether or not decision makers understood the models, were confident in the model results and to identify how the models were used in decision making.
Method: Informed by previously published literature, we developed a theoretical framework on the use of decision models by policy decision-makers. We performed content analysis of documents related to the CTC coverage decision, including the final decision memo and the minutes of the MEDCAC meeting. Using the qualitative method of latent coding we identified themes and adapted our framework to incorporate newly emerging themes. Themes related to perceptions of models included understanding of, confidence in and beliefs about models. Themes related to the use of modeling included focusing or distracting the discussion, influence on the decision and irrelevant to the decision. We used character counts to measure the proportion of text associated with different themes.
Result: MEDCAC meeting minutes revealed that presentation of the models consumed 4% of the meeting and discussion of the models an additional 16%. There was a clear lack of understanding of the models as confusion was expressed much more frequently than understanding. Decision makers expressed a lack of confidence in the models, challenging the key questions addressed by the models, in addition to the inputs and assumptions. For example, decision makers’ were interested in assessing combinations of screening strategies that had not been addressed by the models. The final decision memo devoted 2% of text to emphasizing the value of modeling cost and health impacts, stressing the validity of the CRC models. The memo stated that the models had limited impact on the decision not to cover CTC when compared to other factors.
Conclusion: Limited time devoted to presenting the models may have reduced understanding. Decision makers’ concerns about the questions addressed by the model, in addition to concerns about inputs and assumptions may have diminished confidence in the model results.
See more of: The 32nd Annual Meeting of the Society for Medical Decision Making