Anomaly detection in stream data processing in real time

MAXIM DUNAEV; KONSTANTIN ZAYTSEV; ROMAN ELCHENKOV; DENIS SAVITSKY

Author(s)

MAXIM DUNAEV National Research Nuclear University MEPhI (Moscow Engineering Physics Institute), 31 Kashirskoe Avenue, Moscow, 115409, Russia

KONSTANTIN ZAYTSEV National Research Nuclear University MEPhI (Moscow Engineering Physics Institute), 31 Kashirskoe Avenue, Moscow, 115409, Russia

ROMAN ELCHENKOV National Research Nuclear University MEPhI (Moscow Engineering Physics Institute), 31 Kashirskoe Avenue, Moscow, 115409, Russia

DENIS SAVITSKY National Research Nuclear University MEPhI (Moscow Engineering Physics Institute), 31 Kashirskoe Avenue, Moscow, 115409, Russia

Abstract

The purpose of the present work is to study methods for solving problems of anomaly detection and prediction of time series values when processing streaming data in real-time in a network environment and their improvement. To solve this problem the authors propose a Real-Time K-Means modification with preliminary markup. The effectiveness of the modification is confirmed by comparing it with the frequently used Streaming K-Means from the Apache Spark Mllib library. To solve the problem of predicting time series when processing streaming data in real-time, the authors propose a modification of the autoregression model with a given AR order by adding the inheritance function of the previous values of the time series to it. The results of comparative experiments of the proposed Real-Time AR modification with classical AR confirmed the effectiveness of the modification, which is especially evident in the presence of anomalies in the behavior of the time series. The proposed algorithm modifications allow not only parallelizing calculations using the deferred computing paradigm but also configuring the model fleetingly in the Apache Spark ecosystem. To conduct experiments with algorithms, a dataset was built - a data slice from 1,000 measurements of the Apache Kafka server metrics log with one topic, two producers, and one consumer. Anomalous fragments were artificially added to the dataset, differing by a large number of messages per second and/or message size. The values of the original dataset were normalized and shifted to the average value of the training fetch. Moreover, static and highly correlated metrics were eliminated. The results of the application of the developed algorithms in solving the problems of detecting and predicting the values of time series have shown that even the presence of behavior anomalies does not distort predictions significantly.

Publication Details

Page(s) 3467-3477

DOI DOI not available

Published Journal: Journal of Theoretical and Applied Information Technology, Volume: 100, Issue: 10, Year: 2022

Request the full text from PASTIC

Keywords

machine learning, deep learning, Apache Software Foundation., Logs, Technological Platform