Data Monitoring for Large Scale Public Health Data

Abstract Modern public health data contains information about changes in disease dynamics that can have significant downstream benefits if these phenomena can be identified. However, systemic data quality issues hamper automated analysis of these vast data volumes, and there is now far too much data (3-4 million data points/day) for public health data experts to inspect manually as they may have done in the past. This interdisciplinary thesis addresses practical questions about large-scale data monitoring that impact public health data users and are also reflected in the larger public health community. This work has been deployed for over a year and a half at the Delphi Research Group at Carnegie Mellon University, a national public health data curator, where data reviewers have been able to detect approximately 200 significant outbreaks, data issues, or changes in disease dynamics from 15 million new data points weekly.