In this post, we are going to go over how to diagnose and fix the problem where Kibana is unable to start because of a ‘ConnectionError’ when talking to Elasticsearch using Transport Layer Security (TLS).

I will walk you through how I diagnosed and fixed each problem I came across when trying to fix this problem. Hopefully, it proves useful to fix any problems you are having with Kibana and Elasticsearch with TLS configured.

---

1. SECTION I – The Problem
   1. FIGURE 1 – Kibana Service Status
2. SECTION II – How to Diagnose and Find the Solution
   1. FIGURE 2 – CURL TLS Certificate Error
   2. FIGURE 3 – Successful API CURL

---

SECTION I – The Problem

The problem is caused by an issue with the TLS certificates issued to either the Kibana instance or the Elasticsearch instance which means that they do not trust each other and are unable to communicate using TLS.

Use the following command to verify the status of Kibana:

systemctl status kibana

FIGURE 1 – Kibana Service Status

SECTION II – How to Diagnose and Find the Solution

Now we are going through the steps I took to fix this issue.

To start a very useful command is:

journalctl -u kibana.service

The command above will allow you to see the detailed reasoning why the service cannot be started. In my case, I had a connection error connecting to the Elasticsearch instance.

I want to confirm that the certificate that Elasticsearch was using was correct. Make sure that the firewall on the Elasticsearch machine is open for port 9200 and then browse to it using a normal web browser.

firewall-cmd –add-port=9200/tcp

In my case when I browsed to the Elasticsearch Application Programming Interface (API) "https://<IP>:9200" I had a TLS certificate error. My web browser did not trust the Elasticsearch instance because of a Domain Name System (DNS) name misconfiguration.

I generated a new certificate with the correct DNS names and then applied this to my Elasticsearch instance. Now when I browse to the Elastcisearch API the certificate is trusted.

I restarted the Kibana service however, it still could not start and I was still getting the same ‘ConnectionError’.

systemctl restart kibana

Because my Windows client that I used to browse to the Elasticsearch API trusted the certificate I knew that the issue was local to the Elastic (ELK) stack machine itself.

Now I need to test the TLS certificates internally. In my case, I am running the Elasticsearch and Kibana services on the same machine. To test the TLS certificate internally from the machine where Kibana is running do a CURL to the Elasticsearch web API. Because I am using a custom Certificate Authority (CA) I am going to specify the CA certificate when I run the CURL command

curl –cacert (Path/to/CA/Cert) https://(hostname):9200

As you can see the problem that has been causing this issue is that the TLS certificate is not valid yet. This was caused by a time issue across my Homelabs domain meaning that when the CA issued the TLS certificate for Elasticsearch it was a time ahead of the actual ELK stack machines time.

FIGURE 2 – CURL TLS Certificate Error

With time across the domain sorted, I re-ran the CURL command and the certificate is now valid so I can properly CURL to the Elasticsearch API.

FIGURE 3 – Successful API CURL

Now I can restart Kibana and the service starts as expected because it can communicate with Elasticsearch using the TLS certificates.

systemctl restart kibana