Methods for Evaluating Prevalence Estimates Generated By the NYC Macroscope Electronic Health Record Surveillance System

BACKGROUND:  In the U.S., outpatient uptake of electronic health records now exceeds 70%, offering enormous potential for population health surveillance. The New York City (NYC) Macroscope Electronic Health Record Surveillance System is being developed, as an extension of the NYC Department of Health and Mental Hygiene’s Primary Care Information Project Hub Population Health System (Hub), to monitor the prevalence of important health conditions, risk factors, and the use of preventive services. Methods to validate NYC Macroscope against data obtained from the gold standard 2013 NYC Health and Nutrition Examination Survey (NYC HANES 2013) were piloted using 2012 obesity data.

METHODS:  NYC Macroscope 2012 prevalence estimates for obesity were compared to two reference sources, the 2012 NYC Community Health Survey (CHS 2012), and NYC HANES 2004. Two sets of estimates were used from each reference source, one for the total population and one limited to those reporting visiting a doctor in the previous year. Estimates stratified by age, sex and neighborhood poverty rate were compared. Metrics of goodness-of-fit included mean standardized difference, rho, and mean prevalence ratio.

RESULTS:  NYC Macroscope 2012 estimates of obesity prevalence were largely similar to estimates obtained from the reference surveys and most comparable to estimates for those in care (mean standardized differences for all survey populations < 1.96). NYC Macroscope 2012 prevalence of measured obesity was 29.5% compared to 25.4% of self-reported obesity for the CHS 2012 population in care, and 28.2% prevalence of measured obesity among the NYC HANES 2004 population in care. The correlation between NYC Macroscope 2012 and CHS 2012 in-care population estimates was 0.84, with NYC Macroscope estimates being 21% higher than CHS estimates, on average. NYC Macroscope estimates did not correlate as well with NYC HANES 2004 estimates (r= .72) but were, on average, only 15% higher.

CONCLUSIONS:  Comparison of NYC Macroscope 2012 obesity estimates against two reference data sources suggested that the NYC Macroscope will generate prevalence estimates similar to those from traditional surveillance methods. We expect even better fit for comparisons to NYC HANES 2013 estimates which will be aligned with respect to both timing and mode of measurement. These methods produce metrics that can be used to compare validity across indicators,  to produce overall summary measures of NYC Macroscope performance, and potentially, to compare NYC Macroscope indicator performance to that of EHR-derived indicators from large health systems, practice-based research networks, and other jurisdictions.