Elastic (ELK) Stack Security - Filebeat Logstash SSL Mutual Authentication Example

In previous tutorials we have implemented Spring Boot + ELK stack. Also in another tutorial we had implemented File Beat + ELK Stack. In this tutorial we saw why filebeats are required and how to configure them.

Previously we have secured elasticsearch instance with username and password and also configured SSL, TLS, and HTTPS for it. However it is not possible to secure logstash and filebeat using credentials. Elasticsearch does not provide that option. In this tutorial we will be securing the connection between logstash and filebeat by implementing ssl authentication using certificates. For this tutorial we will be making use of the latest elasticsearch version which is Elasticsearch 8.

Implementation

For the Logstash and Filebeat mutual authentication we will be needing the following certificates -

• ca.crt (Certificate Authority): This file contains the public certificate of the trusted Certificate Authority (CA) that issued the server (Logstash) and client (Filebeat) certificates.
  Both Filebeat and Logstash use this CA certificate to verify the authenticity of each other's certificates during the TLS handshake.
  The CA certificate serves as a trusted root, and all certificates signed by this CA are considered valid and trusted.
• elk.crt (Public Server Certificate): This file contains the public certificate.
  During the TLS handshake, Logstash and Filebeat present this certificate to authenticate their identity to each other.
  The CA certificate (ca.crt) is used to verify the validity and authenticity of the elk.crt server certificate.
• elk.key (Private Key): This file contains the private key corresponding to the elk.crt server certificate.
  The private key is used to prove its ownership of the elk.crt certificate during the TLS handshake.
  Private key is used to encrypt a part of the TLS handshake data, which can then be decrypt using the public key from the elk.crt certificate.

Following is how the Mutual TLS between Filebeat and Logstash will work.

• Filebeat Initiates Connection:
  Filebeat initiates a connection to Logstash on localhost:5044 over an encrypted SSL/TLS channel. During the TLS handshake, Filebeat sends its client certificate elk.crt to Logstash.
• Logstash Verifies Filebeat's Certificate:
  Logstash receives Filebeat's client certificate. Logstash uses the trusted CA certificate - ca.crt to validate the authenticity of Filebeat's client certificate. If the validation is successful, Logstash trusts that the client (Filebeat) is authentic.
• Logstash Authenticates Itself to Filebeat:
  Logstash sends its server certificate elk.crt to Filebeat.
• Filebeat Verifies Logstash's Certificate:
  Filebeat receives Logstash's server certificate. Filebeat uses the trusted CA certificate - ca.crt to validate the authenticity of Logstash's server certificate. If the validation is successful, Filebeat trusts that the server (Logstash) is authentic.
• Secure Communication Established:
  Filebeat and Logstash then encrypt the data using the elk.key as part of the TLS handshake. This encrytped data can be decrypted using public elk.crt Filebeat can now start sending log data to Logstash over this secure channel.

Next let us begin with the setup. We will first set up logstash and filebeat without any security.

Logstash

Download Logstash.

Next unzip the downloaded logstash executable. In the logstash folder create a logstash.conf file with the following configuration -

input {
  beats {
    port => 5044
  }
}
output {
 
 stdout {
  codec => rubydebug
 }
}

The provided Logstash configuration sets up an input to receive data from Filebeat and an output to print the received data to the console (stdout) using the rubydebug codec.
Next start logstash using following command

logstash.bat -f logstash.conf

Filebeat

Download Filebeat.

Next unzip the downloaded filebeat executable. In the filebeat folder modify the filebeat.yml file with the following configuration -

filebeat.inputs:
- type: log
  paths:
    - E:/logs/*.log   

output.logstash:
  hosts: ["localhost:5044"]

Next start filebeat as follow-

filebeat.exe -c filebeat.yml

We can see that logstash recieves the data sent by filebeat.

Security

To implement security we will also need to download elasticsearch. Using the elasticsearch cert-util we will be generating the required certificates Go to the elasticsearch downloads page. Click on the Windows button to download the latest elasticsearch installable.

Unzip the downloaded elasticsearch executable. Next we will be generating the CA certificate as follows -

 elasticsearch-certutil ca --pem --out E:\elk\cert\ca.zip
 

Using this CA certificate we will be generating the node certificates. For this we will also need to specifies a list of instances (nodes) for which certificates should be generated. This is done using the following instances.yml file. The instances.yml file allows you to specify multiple instances if you need to generate certificates for a multi-node Elasticsearch cluster. Each instance entry in the file will result in a separate certificate being generated, with the specified DNS names or IP addresses included as SANs.

instances:
  - name: 'elk'
    dns: [ 'localhost' ]

elasticsearch-certutil cert --pem --ca-cert E:\elk\cert\ca\ca.crt --ca-key E:\elk\cert\ca\ca.key --in E:\elk\cert\instances.yml --out E:\elk\cert\elk-cert.zip --days 365

Finally using the OpenSSL command-line tool we will convert the elk.key private key file to the PKCS#8 format.

openssl pkcs8 -in E:\elk\cert\elk-cert\elk\elk.key -topk8 -nocrypt -out E:\elk\cert\elk-cert\elk\elk.pkcs8.key

Let us now make use of the generated certificates to implement mutual TLS authentication.
Modify the filebeat config file as follows-

filebeat.inputs:
- type: log
  paths:
    - E:/logs/*.log   

output.logstash:
  hosts: ["localhost:5044"]
  ssl.certificate_authorities: ["E:/elk/cert/ca/ca.crt"]
  ssl.certificate: "E:/elk/cert/elk-cert/elk/elk.crt"
  ssl.key: "E:/elk/cert/elk-cert/elk/elk.pkcs8.key"

Next modify the logstash config file-

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => "E:\elk\cert\ca\ca.crt"
    ssl_certificate => "E:\elk\cert\elk-cert\elk\elk.crt"
    ssl_key => "E:\elk\cert\elk-cert\elk\elk.pkcs8.key"
  }
}
output {
 
 stdout {
  codec => rubydebug
 }
}

These are the only changes required. The connection between filebeat and logstash is now secure.

See Also
Spring Boot Hello World Application- Create simple controller and jsp view using Maven
Spring Boot Tutorial-Spring Data JPA
Spring Boot + Simple Security Configuration
Pagination using Spring Boot Simple Example
Spring Boot + ActiveMQ Hello world Example
Spring Boot + Swagger Example Hello World Example
Spring Boot + Swagger- Understanding the various Swagger Annotations
Spring Boot Main Menu
Spring Boot Interview Questions