Leveraging Anomaly Detection for Proactive Application Monitoring

Anomaly detection is one of the popular research fields in Machine Learning. Also, this is one of the key techniques in system and application monitoring in Industry. Anomaly detection comprises of outlier detection and identifying novelty from the data -

  • PDF / 330,651 Bytes
  • 6 Pages / 439.37 x 666.142 pts Page_size
  • 42 Downloads / 232 Views

DOWNLOAD

REPORT


Abstract. Anomaly detection is one of the popular research fields in Machine Learning. Also, this is one of the key techniques in system and application monitoring in Industry. Anomaly detection comprises of outlier detection and identifying novelty from the data - it is a process to understand the deviation of an observation from existing observations [12] and identifying the new observations. Carrying out anomaly detection in an enterprise application is a challenge as there are complex processes to gather and analyze functional and non-functional logs of unlabeled data. In this paper we are proposing an unsupervised learning process with log featurization incorporating time window to detect outliers and novel errors from enterprise application logs. Keywords: Anomaly detection · Outlier detection · Novelty detection

1 Introduction Application monitoring with a real-world dataset is a complex task as huge quantities and a variety of log information are getting generated in every minute. The log information can be functional (application functionality logs) and non-functional (system operation logs, performance logs, security logs etc.), and these are unlabeled information. There exist plenty of anomaly detection algorithms in different categories for addressing the need. The log information is dynamic in nature as it is generated during a process journey and depends upon the data values in each step. And new logs will also be added during each cycle of application development. Rule based log capturing and anomaly detection methods will have enormous issues in these situations, as rules cannot cope with these frequent and dynamic changes. Our proposed method can capture the log in the same format and can also find the outliers and novel errors in the logs. We consider outliers as the errors which are deviant from the normal concentrated observation of data. We also define novel errors as errors which are not seen in the training data. In this paper, we address the problem of anomaly detection from the log dataset by converting the log information to appropriate featurization and applying different unsupervised anomaly detection algorithms. The end-to-end strategic approach to tackle outlier and novelty detection within enterprise application logs is introduced in this paper. The paper is organized as follows: In Sect. 2, we will highlight the related work in this area. Section 3 describes the proposed approach to the problem, and Sect. 4 contains © Springer Nature Switzerland AG 2020 M. Bramer and R. Ellis (Eds.): SGAI-AI 2020, LNAI 12498, pp. 380–385, 2020. https://doi.org/10.1007/978-3-030-63799-6_29

Leveraging Anomaly Detection for Proactive Application Monitoring

381

experiment results and benefits. Section 5 concludes the paper and presents future work. Acknowledgements for the paper follow.

2 Related Works The mechanism to capture operational logs has been in existence since a long time. But most of the techniques were using a regex expression to parse the log file [8–10] to build the features from the