Variable-Length Anomalous Intervals in Data Streams

Anomaly-management tools monitor data streams in complex systems, e.g., tracking weather parameters collected from sensors in a region or inspecting resource usage of machines in a data center. Their goal is to detect outliers and abnormal behavior of the system, including a burst of outliers, which is often the result of a major event, like a hurricane or a crash of machines in a data center. An interval alert is an alert raised for such a burst of outliers. In data pipelines, such alerts indicate time intervals during which major data quality issues may occur. This allows alerting downstream applications on potential events and data quality risks. In this paper we present Ourea, a novel unsupervised learning method that computes interval alerts by identifying anomalous time intervals of variable lengths in data streams. Ourea raises interval alerts on significant events, based on a concentration of outliers detected in the raw data stream and in high-level aggregate views. It uses Kernel Density Estimate (KDE) to identify the most significant time intervals and their boundaries, can issue preliminary alerts early before intervals end, and minimizes the number of raised alerts by alerting only on significant ones. Extensive experiments over real and synthetic data demonstrate the effectiveness and scalability of Ourea and show that for detecting variable-length anomalous intervals in data streams, our algorithm is more accurate and more efficient than state-of-the-art methods.