Enhanced Security Using Elasticsearch and Machine Learning
The purpose of this paper is to highlight how can Elasticsearch be used to enhance the security of your applications and your cloud infrastructure by combining intrusion detection systems with machine learning techniques in order to detect possible attack
- PDF / 1,828,794 Bytes
- 11 Pages / 439.37 x 666.142 pts Page_size
- 48 Downloads / 546 Views
Abstract. The purpose of this paper is to highlight how can Elasticsearch be used to enhance the security of your applications and your cloud infrastructure by combining intrusion detection systems with machine learning techniques in order to detect possible attacks. It will cover the setup and configuration of a test environment for anomaly detection and network security alerting using Elasticsearch as the core for storing data. Snort is used for monitoring, alongside system and network analytics collected via Metricbeat and Packetbeat. Built-in machine learning jobs from Elastic will be used to find disturbances in the normal operation of the devices. To create a baseline dataset the Damn Vulnerable Web application is used to generate analytics and alerts upon exploiting the vulnerabilities exposed. Keywords: OpenStack · Elasticsearch · Logstash · Kibana · Metricbeat · Packetbeat · Snort · DVWA · Hierarchical Temporal Memory
1 Introduction With the continuous increase of cloud-based services and IoT connected devices there is also a need for better availability and reachability between those internet nodes. Migrating your applications and services to a scalable system, like the cloud environment will boost your performance, but will also increase your product’s attack surface, generated by the infrastructure’s vulnerabilities or by other applications running on the same server. To prevent such scenarios powerful and costly equipment is necessary, or software capable of detecting threats in real-time. Common defence approaches include signature-based intrusion detection systems or web application firewalls, which can achieve real-time monitoring and analysis of network traffic, application and system logs. Most tools also provide file integrity checks, configuration and permissions assessments, policy violation and malicious behaviour detection, active threat prevention and attack surface reduction. But, because all of these are based on known vulnerabilities and exploits, hybrid solutions appeared that are enhancing those mechanisms with supervised or unsupervised machine learning techniques to better detect incoming attacks. The main purpose of this research is to study popular security tools and how they integrate with Elasticsearch, and how can machine learning methods can be used to detect anomalies inside an infrastructure. The architecture proposed consists of three © Springer Nature Switzerland AG 2020 K. Arai et al. (Eds.): SAI 2020, AISC 1230, pp. 244–254, 2020. https://doi.org/10.1007/978-3-030-52243-8_19
Enhanced Security Using Elasticsearch and Machine Learning
245
virtual machines with two of those being Elasticsearch nodes and anomaly detectors, and the last one will act as both a physical device and an application container. Its normal operation parameters will be monitored with Metricbeat and Packetbeat and the network traffic will be analysed by Snort, configured with the community rules. The Damn Vulnerable Web Application will also be installed and used as a threat generator, to easily gen
Data Loading...
