PURPOSE The NHS Value of Health Panel is a pilot project investigating the feasibility, acceptability, reliability and validity of establishing and maintaining a panel of members of the public with the purpose of obtaining utility data in relation to specific health states using the internet. The panel (n=112) considered 102 health state descriptions, presented in 18 “sets” (a range of disease states and EQ5D states). The purpose of this element of the study was to investigate construct validity.

METHODS Preferences were elicted using a titration variant of the standard gamble with graphical risk representation. Construct validity was assessed using logical consistency. We identified pairs of health states, within the conditions depicted, which were unambiguously ordered in severity and noted whether the utilities obtained from respondents showed (a) appropriate order (b) tied values or (c) reversed order. Univariate and multivariate analyses of potential determinants of response (by person and state-pair) were carried out.

RESULTS 82 logically ordered pairs of health states were identified, on which 2,936 values were obtained. Overall 322 (11%) of values were not logically consistent, with 187 (6.4%) ties and 135 (4.6%) order reversals. By respondent, the proportion of values which were not logically consistent was highly positively skewed (mean 12.3%, median 8.9%, range 0-75%). By set of health state description, the proportion of illogical responses varied between 0% (EQ5D) and 41% (COPD). ANOVA was significant across sets. In 14/18 sets, the proportion of tied responses was greater than the proportion of order reversals. There was no significant association with age, sex, socioeconomic status, education level or income. For health state descriptions, the association with Flesch reading score approached significance, but no other association with health state-pair characteristics.

CONCLUSIONS Although overall rates of inconsistency were similar to previous studies, there was considerable variation by health state and person. Further research, using a larger panel, is justified to investigate the determinants of logical response further and the distinction between tied and reversed values deserves more attention. Explanatory IRT modelling has the potential to investigate the effect of both item and person sides of such data, though larger sample sizes are required.