Benefits of Hindsight: Early Results and Lessons from the Evaluation of Massachusetts' Infectious Disease Surveillance System

BACKGROUND: Beginning in late 2011, the Massachusetts Department of Public Health Bureau of Infectious Disease (BID) began a comprehensive evaluation of their electronic web-based surveillance platform, the Massachusetts Virtual Epidemiologic Network (MAVEN). Representatives from key programmatic areas formed a working group to evaluate the quality of the surveillance system and its data, identify problems, and make improvements to the system internally and externally. The working group’s wide array of projects includes timeliness of reporting and validation of a subset of disease reports. We first focused on the completeness of critical case record elements, the appropriate assignment of jurisdiction (for notification purposes), and the proportion of cases adequately investigated. 

METHODS: MAVEN records on 40 reportable diseases from 2007 through 2011 were analyzed. The completeness of basic demographic information was quantified. Core functions of the surveillance office, including notification of local health departments regarding cases in their jurisdictions and elicitation of missing address information were examined. The proportion of cases for which an investigation was started, and for which an investigation was appropriately completed, was quantified.

RESULTS: There were 140,537 individual disease events during the study period. We found that appropriate jurisdiction was assigned for 81-99%, depending on the disease category (i.e. immediate, routine, or clinician-based). We confirmed that the surveillance office carries out appropriate notifications for the large majority of reported diseases. A large proportion of cases had missing race and/or ethnicity information; the proportion missing varied by disease, reporting source, and investigator (e.g. race was missing for <2% of tuberculosis cases, compared to 30% of hepatitis A cases). With respect to the proportion of cases with investigations started and adequately investigated, much time and effort was spent defining the terms per disease; substantial variation in results was identified dependent on the disease and investigator.  

CONCLUSIONS: While this long-term evaluation project is still underway, we have identified and responded to numerous large systematic issues including: changes to MAVEN to better capture ethnicity information and more clear documentation for why a case investigation was not completed. We have learned numerous lessons regarding the evaluation of a system like MAVEN and we believe our findings will be of value to other MAVEN users and jurisdictions with similar systems in the US.