METHODS: We used an inter-rater reliability framework to test the consistency of abuse, neglect, and negligence classifications made through the CDR process. All children born in 2009-2010 who died in Alaska through 2014 were eligible for inclusion (n=116); after exclusions 101 cases were reviewed by two CDR review panels. Approximately half of the cases were included with normal monthly reviews, the rest were reviewed at an annual committee review. Cross-classification summary tables were constructed and weighted Kappa, percent agreement, rater bias coefficients, and Chi Square test were calculated.
RESULTS: Total percent agreement between panels for maltreatment was 64.7% with a weighted Kappa of 0.61 (95%CI 0.51, 0.70). The percent agreement was 69.3% for abuse, 64.4% for neglect, and 60.4% for negligence. Agreement was lowest among deaths coded as “Natural” (0.47), among unmarried mothers (0.47), and infant deaths (0.48). By combining the Yes and Yes probably categories, both panels uniquely identified 37 maltreatment-related fatalities and incidence of 1.6 (95%CI 1.1, 2.2) per 1,000 live births. The maltreatment-related incidence varies from 1.2 (95%CI 0.8, 1.7) using only cases where both panels agree, to 2.1 (95%CI 1.5, 2.8) by combining both panel classifications. Unlike neglect and negligence classifications that had large variation in incidence estimates from cases where both panels agree to combined estimates, abuse estimates were much more stable
CONCLUSIONS: This is the first known study of CDR team maltreatment classification reliability. Allowing for increased variability due to ambiguity of information through polytomous maltreatment definitions results in sub-optimal reliability. Poor reliability impacts public health efforts to prevent maltreatment-related mortality. Although tiered CDR definitions allow for increased detection of cases, systematic decision metrics are needed to improve reliability and stability of estimates.