Background: Public disclosure of hospital-acquired infection (HAI) rates is gaining momentum across the United States. However, traditional manual surveillance for HAI is hindered by the fact that National Nosocomial Infection Surveillance (NNIS) system criteria are complex and subjective, and are far from a diagnostic "gold standard". This raises concerns about the reliability of these criteria, particularly as they are applied by different individuals across different institutions. Simplified, objective criteria based only on microbiologic data may be a less valid, but potentially more reliable system for comparing institutional infection rates.

Methods: We developed an agent-based model to examine the effect of validity and reliability on the measurement of catheter-related bloodstream infections (CRBSI) in a simulated 12-bed hospital intensive care unit (ICU). Traditional (clinical-based) surveillance was performed by simulated surveyors applying NNIS criteria; their reliability at interpreting subjective criteria was modeled using concepts derived from signal detection theory. Algorithmic surveillance was performed by applying simplified criteria to microbiologic data based on previously published work. Reliability of the simulated surveyors was assessed by altering their accuracy and their variance in specificity in a sensitivity analysis. Preservation of rank order was assessed using Kendall's Tau method.

Results: On average, the clinical criteria was more accurate at estimating the true CRBSI rate than the simple criteria (4.60+/-1.31 vs. 7.51+/-1.54 infections/1000 catheter-days, compared with a true rate of 4.91+/-1.93 infections/1000 catheter-days), owing mostly to a higher average specificity (92.2% vs. 81.5% for simple criteria). However, as rater accuracy and variance in specificity were varied, and across a plausible range of inter-rater reliabilities (mean 0.53, range 0.33-0.72), the accuracy of clinical criteria approaches, and is often worse than, simple criteria. In addition, under the majority of conditions and scenarios, ecologic correlation (i.e., the accurate ranking of CRBSI rates across institutions) is higher for simple criteria than clinical criteria, even in situations where individual accuracy is higher for clinical criteria.

Conclusions: Simplified CRBSI surveillance criteria, which remove subjectivity, are more reliable than traditional clinical criteria at accurately identifying the true differences in CRBSI rates between institutions. Thus, the more reliable simple criteria are most likely better suited for measuring and publicly reporting institutional CRBSI rates for comparative purposes.