Developing Measures to Evaluate Birth Registration E-Learning in New York City, 2012

BACKGROUND: Variation in the procedures, background, resources, and training of staff that collect and enter information in the New York City (NYC) electronic birth registration system (EBRS) impacts birth data quality. As a result, the NYC Health Department developed an Internet-based training that can be self-administered at birthing facilities. Though evaluation is critical for training development, few examples exist in this setting. Study objectives are to describe selected evaluation measures, data collection plans, and lessons learned.

METHODS: We reviewed the literature for examples of training evaluations and unpublished resources from other vital statistics jurisdictions. Second, we interviewed NYC Health Department staff and experts in evaluation, vital event registrations systems, and Internet-based trainings. Third, we developed research questions based on the training logic model. Finally, birth records from 2010–2012 and a 2009 survey of NYC hospital staff (n = 39) were analyzed to identify variables that were poor quality, difficult to collect, or important for birth certificate issuance or maternal child health research.

RESULTS: Three evaluation measures were chosen: training efficacy, functionality, and birth data quality. We developed pre/post-training surveys that measured efficacy by assessing user reaction, knowledge (both content and perceived knowledge gains), attitudes (i.e., self-efficacy, satisfaction), behaviors, and perceived workplace value. The post-training survey also included five questions that measured functionality by asking about perceptions of training content, accessibility (i.e., literacy level), usability, and resource availability (i.e., computer access). To measure data quality, we selected seven frequently used variables and created a composite measure of the percent of unknown and/or conflicting values (e.g., inconsistent date of last live birth and number of previous live births). We also developed four specialized EBRS reports that summarize the number of duplicate birth records, errors to mother’s address, changes to child’s name, and changes to marital status pre and post-training.

CONCLUSIONS: Birth data are essential for administrative, clinical, research, and programmatic purposes, yet few resources for implementing or evaluating data quality improvement programs exist. Ultimately, measures for this study were developed using literature from other disciplines and stakeholder knowledge and balance the desire for internal validity with feasibility this setting. Our findings can be used to inform local and national efforts to develop and evaluate trainings for vital event registration. Lessons are also relevant for surveillance systems that rely on administrative or clinical reporting. Preliminary survey results and data quality measures will also be presented.