Methods: We conducted a systematic literature search for process measures utilized for assessing the quality of colorectal cancer (CRC)care, and developed a rating scale for evaluating the quality of the CPMs. Rating was conducted by two investigators in two steps: (1) Each CPM, in each study, was scored for importance, usability, and scientific acceptability (precise specifications, reliability, validity, adaptability) on a 1-5 scale (1=poor, 5=ideal). (2) Each CPM was rated, across all studies, based on importance (and usability), scientific acceptability and level of field-testing on a scale of 1-5. We designated this as the “IST rating scale”. The “I” rating was determined by calculating the average of the scores assigned for Importance and Usability, and considering the highest score. The “S” rating was determined by calculating the average of the scores assigned for each of the five criteria for scientific acceptability and considering the highest average score. The “T” rating was determined by considered two aspects of the measure: number of studies that utilized the measure, and the S rating of the measure. We calculated the inter-rater reliability for assigning I, S and T ratings.

Results: We created an inventory of CPMs for CRC care, and developed a rating scale for the measures. The inter-rater reliability for I rating was 0.36, for S rating it was 0.76, and for T rating it was 0.88.

Conclusions: The IST rating scale provides a succinct summary of the most relevant information regarding CPMs and presents it in a manner that is germane to making decisions regarding the their use. For example a policy maker may opt to implement CPMs with high I, S and T rating. Provider organizations could implement those with a high S and T rating, measure developers would further develop measures with a low S rating, and researchers could field-test CPMs with a low T rating. This is, to our knowledge, the first scale that has been developed for rating CPMs. We utilized this rating scheme for process measures, and it can be extrapolated to structure as well as outcome measures.

The project was funded by the Agency for Healthcare Research and Quality)