How to configure SSL/TLS?

#1

Hi,

we’re currently using the RPM’s for installing open distro elasticsearch and running into a few issues trying to replace the certificates.

We already own certificates, so do not want to generate new ones.

Tried a lot of things, but basically from what I get it should boil down to replacing the default kirk certificates. In order to keep this a bit short, I’ll only describe the short track attempt here. Basically I just replace, under /etc/elasticsearch, kirk.pem, kirk-key.pem and ca-cert.pem and with our certificate, key and intermediate.

Then remove elasticsearch.keystore.

chmod g+w /etc/elasticsearch
(can’t recreate the keystore otherwise - has no write permissions – think that’s an issue btw ;))

restart elasticsearch by ‘systemctl restart elasticsearch’ and it’ll log:

Throws no errors, but ports aren’t opened (and thus does throw errors on 9200 not being available):

Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Stopping Elasticsearch…
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Stopping Opendistro for Elasticsearch Performance Analyzer…
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Started Opendistro for Elasticsearch Performance Analyzer.
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Starting Opendistro for Elasticsearch Performance Analyzer…
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Started Elasticsearch.
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Starting Elasticsearch…
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.security.policy: error adding Entry:
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.net.MalformedURLException: unknown protocol: jrt
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.security.policy: error adding Entry:
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.net.MalformedURLException: unknown protocol: jrt
Mar 25 16:34:58 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:34:58.776 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:00 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:00Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:00 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:00Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
Mar 25 16:35:01 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:01.266 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:02 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:02Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:02 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:02Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
^C
[root@ip-10-150-33-134 elasticsearch]# Mar 25 16:35:03 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:03.767 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:05 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:05Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:05 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:05Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
Mar 25 16:35:06 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:06.268 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:07 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:07Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:07 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:07Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
Mar 25 16:35:08 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:08.770 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:10 ip-10-150-33-134 kibana[2206]: {“type”:“log”,"@timestamp":“2019-03-25T16:35:10Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}

Tried many different things btw, this is just the short version. Results were similar with the longer attempts.

Nothing which indicates where it goes wrong under /var/log/elasticsearch either btw. systemctl seems to think everything has started well, but unfortunately that’s not the case.

Our official cert has ‘extended key usage’ set, but does support client & server authentication:

    X509v3 extensions:
        X509v3 Basic Constraints: 
            CA:FALSE
        X509v3 Authority Key Identifier: 
            keyid:91:19:62:AD:5B:17:A7:30:FB:F0:DE:39:25:B1:BD:8C:B9:B8:51:27

        Authority Information Access: 
            CA Issuers - URI:http://trust.quovadisglobal.com/qvsslg2.crt
            OCSP - URI:http://ocsp.quovadisglobal.com

        X509v3 Subject Alternative Name: 
            DNS:elasticsearch.somedomain.tld, DNS:kibana.somedomain.tld
        X509v3 Certificate Policies: 
            Policy: 1.3.6.1.4.1.8024.0.2.100.1.1
              CPS: http://www.quovadisglobal.com/repository

        **X509v3 Extended Key Usage: **

** TLS Web Client Authentication, TLS Web Server Authentication**
X509v3 CRL Distribution Points:

            Full Name:
              URI:http://crl.quovadisglobal.com/qvsslg2.crl

        X509v3 Subject Key Identifier: 
            1A:1F:D2:7F:9E:76:06:3B:D6:33:92:45:A2:53:3F:7C:6D:74:CB:E6
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        CT Precertificate SCTs: 
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : 56:14:06:9A:2F:D7:C2:EC:D3:F5:E1:BD:44:B2:3E:C7:
                            46:76:B9:BC:99:11:5C:C0:EF:94:98:55:D6:89:D0:DD
                Timestamp : Mar 18 08:09:48.267 2019 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:46:02:21:00:A8:EE:0F:00:83:AB:68:DA:5E:BE:B1:
                            9A:DA:28:C0:73:B5:32:B3:86:4A:E7:FF:A3:A1:28:28:
                            D7:42:40:AF:91:02:21:00:B4:55:22:8A:CE:5A:3F:DB:
                            9C:2F:1A:8B:57:EF:94:30:E5:DA:C4:05:90:61:F8:E1:
                            39:69:0F:43:D8:DD:F8:8A
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
                            15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
                Timestamp : Mar 18 08:09:48.372 2019 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:45:02:21:00:EC:57:71:A3:9A:64:96:5B:50:06:4C:
                            F1:B3:67:0F:FF:29:A4:68:6C:0E:51:09:33:76:05:8A:
                            C4:0E:E3:0A:C7:02:20:70:4C:7D:D3:BE:04:1B:2E:DD:
                            37:45:E7:63:53:BD:19:55:C9:B6:C3:2C:88:BA:22:BF:
                            7F:CD:FD:41:C7:EF:5B
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : BB:D9:DF:BC:1F:8A:71:B5:93:94:23:97:AA:92:7B:47:
                            38:57:95:0A:AB:52:E8:1A:90:96:64:36:8E:1E:D1:85
                Timestamp : Mar 18 08:09:48.257 2019 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:46:02:21:00:EE:48:3F:FF:9A:3D:5C:24:63:0A:B9:
                            4E:9D:35:FC:6A:6D:62:36:B2:0D:79:D5:5A:D1:94:1F:
                            8C:10:E8:61:31:02:21:00:E0:C5:79:2C:1B:5B:10:55:
                            1C:DF:3C:5F:4B:1C:6C:0A:B9:63:95:40:15:4C:2F:42:
                            E9:CC:27:10:37:68:12:F8

Any suggestions on how to debug/proceed? Would be much obliged :).

#2

Key was in a format that has header line:
BEGIN RSA PRIVATE KEY
converted it to format
BEGIN PRIVATE KEY

and the issues are gone.