Hi,
we’re currently using the RPM’s for installing open distro elasticsearch and running into a few issues trying to replace the certificates.
We already own certificates, so do not want to generate new ones.
Tried a lot of things, but basically from what I get it should boil down to replacing the default kirk certificates. In order to keep this a bit short, I’ll only describe the short track attempt here. Basically I just replace, under /etc/elasticsearch, kirk.pem, kirk-key.pem and ca-cert.pem and with our certificate, key and intermediate.
Then remove elasticsearch.keystore.
chmod g+w /etc/elasticsearch
(can’t recreate the keystore otherwise - has no write permissions – think that’s an issue btw ;))
restart elasticsearch by ‘systemctl restart elasticsearch’ and it’ll log:
Throws no errors, but ports aren’t opened (and thus does throw errors on 9200 not being available):
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Stopping Elasticsearch…
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Stopping Opendistro for Elasticsearch Performance Analyzer…
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Started Opendistro for Elasticsearch Performance Analyzer.
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Starting Opendistro for Elasticsearch Performance Analyzer…
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Started Elasticsearch.
Mar 25 16:34:57 ip-10-150-33-134 systemd[1]: Starting Elasticsearch…
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.security.policy: error adding Entry:
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.net.MalformedURLException: unknown protocol: jrt
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.security.policy: error adding Entry:
Mar 25 16:34:58 ip-10-150-33-134 elasticsearch[13774]: java.net.MalformedURLException: unknown protocol: jrt
Mar 25 16:34:58 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:34:58.776 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:00 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:00Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:00 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:00Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
Mar 25 16:35:01 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:01.266 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:02 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:02Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:02 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:02Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
^C
[root@ip-10-150-33-134 elasticsearch]# Mar 25 16:35:03 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:03.767 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:05 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:05Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:05 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:05Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
Mar 25 16:35:06 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:06.268 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:07 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:07Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Mar 25 16:35:07 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:07Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“No living connections”}
Mar 25 16:35:08 ip-10-150-33-134 performance-analyzer-agent-cli[13765]: 16:35:08.770 [Thread-1] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.reader.ClusterLevelMetricsReader - Skip parsing. Number of lines: 1.
Mar 25 16:35:10 ip-10-150-33-134 kibana[2206]: {“type”:“log”,“@timestamp”:“2019-03-25T16:35:10Z”,“tags”:[“warning”,“elasticsearch”,“admin”],“pid”:2206,“message”:“Unable to revive connection: NOT-ALLOWED-TO-POST-LINKS-BUT-THIS-READ-HTTPS-LOCALHOST:9200/”}
Tried many different things btw, this is just the short version. Results were similar with the longer attempts.
Nothing which indicates where it goes wrong under /var/log/elasticsearch either btw. systemctl seems to think everything has started well, but unfortunately that’s not the case.
Our official cert has ‘extended key usage’ set, but does support client & server authentication:
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Authority Key Identifier:
keyid:91:19:62:AD:5B:17:A7:30:FB:F0:DE:39:25:B1:BD:8C:B9:B8:51:27
Authority Information Access:
CA Issuers - URI:http://trust.quovadisglobal.com/qvsslg2.crt
OCSP - URI:http://ocsp.quovadisglobal.com
X509v3 Subject Alternative Name:
DNS:elasticsearch.somedomain.tld, DNS:kibana.somedomain.tld
X509v3 Certificate Policies:
Policy: 1.3.6.1.4.1.8024.0.2.100.1.1
CPS: http://www.quovadisglobal.com/repository
**X509v3 Extended Key Usage: **
** TLS Web Client Authentication, TLS Web Server Authentication**
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.quovadisglobal.com/qvsslg2.crl
X509v3 Subject Key Identifier:
1A:1F:D2:7F:9E:76:06:3B:D6:33:92:45:A2:53:3F:7C:6D:74:CB:E6
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1(0)
Log ID : 56:14:06:9A:2F:D7:C2:EC:D3:F5:E1:BD:44:B2:3E:C7:
46:76:B9:BC:99:11:5C:C0:EF:94:98:55:D6:89:D0:DD
Timestamp : Mar 18 08:09:48.267 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:A8:EE:0F:00:83:AB:68:DA:5E:BE:B1:
9A:DA:28:C0:73:B5:32:B3:86:4A:E7:FF:A3:A1:28:28:
D7:42:40:AF:91:02:21:00:B4:55:22:8A:CE:5A:3F:DB:
9C:2F:1A:8B:57:EF:94:30:E5:DA:C4:05:90:61:F8:E1:
39:69:0F:43:D8:DD:F8:8A
Signed Certificate Timestamp:
Version : v1(0)
Log ID : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
Timestamp : Mar 18 08:09:48.372 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:EC:57:71:A3:9A:64:96:5B:50:06:4C:
F1:B3:67:0F:FF:29:A4:68:6C:0E:51:09:33:76:05:8A:
C4:0E:E3:0A:C7:02:20:70:4C:7D:D3:BE:04:1B:2E:DD:
37:45:E7:63:53:BD:19:55:C9:B6:C3:2C:88:BA:22:BF:
7F:CD:FD:41:C7:EF:5B
Signed Certificate Timestamp:
Version : v1(0)
Log ID : BB:D9:DF:BC:1F:8A:71:B5:93:94:23:97:AA:92:7B:47:
38:57:95:0A:AB:52:E8:1A:90:96:64:36:8E:1E:D1:85
Timestamp : Mar 18 08:09:48.257 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:EE:48:3F:FF:9A:3D:5C:24:63:0A:B9:
4E:9D:35:FC:6A:6D:62:36:B2:0D:79:D5:5A:D1:94:1F:
8C:10:E8:61:31:02:21:00:E0:C5:79:2C:1B:5B:10:55:
1C:DF:3C:5F:4B:1C:6C:0A:B9:63:95:40:15:4C:2F:42:
E9:CC:27:10:37:68:12:F8
Any suggestions on how to debug/proceed? Would be much obliged :).