Usage of kibana svc

General Feedback
1

Am I to use 5601 or 443 as the target port? It seems like 443 is the answer:

$ kubectl port-forward service/whc01elastic-opendistro-es-kibana-svc 5602:5601
error: Service whc01elastic-opendistro-es-kibana-svc does not have a service port 5601
$ kubectl port-forward service/whc01elastic-opendistro-es-kibana-svc 8443:443
Forwarding from 127.0.0.1:8443 -> 5601
Forwarding from [::1]:8443 -> 5601

However, I only get Kibana server is not ready yet, looking at its logs they are filled with:

{"type":"log","@timestamp":"2021-06-27T16:39:38Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}

whereas the very head of the logs look like this:

{"type":"log","@timestamp":"2021-06-27T16:38:43Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2021-06-27T16:38:45Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2021-06-27T16:38:48Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2021-06-27T16:38:50Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2021-06-27T16:06:32Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"telemetryManagementSection\" has been disabled since the following direct or transitive dependencies are missing or disabled: [telemetry]"}
{"type":"log","@timestamp":"2021-06-27T16:06:32Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"newsfeed\" is disabled."}
{"type":"log","@timestamp":"2021-06-27T16:06:32Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"telemetry\" is disabled."}
{"type":"log","@timestamp":"2021-06-27T16:06:32Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-06-27T16:06:32Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
{"type":"log","@timestamp":"2021-06-27T16:06:32Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
{"type":"log","@timestamp":"2021-06-27T16:06:33Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [46] plugins: [opendistroAlertingKibana,usageCollection,telemetryCollectionManager,kibanaUsageCollection,securityOss,mapsLegacy,kibanaLegacy,share,legacyExport,embeddable,esUiShared,expressions,data,home,apmOss,console,management,indexPatternManagement,advancedSettings,savedObjects,opendistroSecurityKibana,opendistroIndexManagementKibana,opendistroAnomalyDetectionKibana,dashboard,opendistroNotebooksKibana,visualizations,visTypeVega,visTypeTimelion,timelion,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,opendistroGanttChartKibana,visualize,opendistroReportsKibana,opendistroQueryWorkbenchKibana,charts,visTypeTimeseries,visTypeVislib,visTypeMetric,visTypeTagcloud,discover,savedObjectsManagement,bfetch]"}
{"type":"log","@timestamp":"2021-06-27T16:06:34Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-06-27T16:06:34Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2021-06-27T16:06:34Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2021-06-27T16:06:36Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: Client network socket disconnected before secure TLS connection was established"}

k describe svc whc01elastic-opendistro-es-kibana-svc
Name:              whc01elastic-opendistro-es-kibana-svc
Namespace:         default
Labels:            app=whc01elastic-opendistro-es
                   app.kubernetes.io/managed-by=Helm
                   chart=opendistro-es-1.13.1
                   heritage=Helm
                   release=whc01elastic
                   role=kibana
Annotations:       meta.helm.sh/release-name: whc01elastic
                   meta.helm.sh/release-namespace: default
Selector:          role=kibana
Type:              ClusterIP
IP:                10.100.145.216
Port:              kibana-svc  443/TCP
TargetPort:        5601/TCP
Endpoints:         10.244.2.66:5601
Session Affinity:  None
Events:            <none>

opendistro installed by the helm chart:

helm install whc01elastic --values=/values/opendistro-values.yaml .

the only thing I’ve changed from the opendistro-1.13.1 is to assign the storageClass:

$ diff opendistro-values.yaml ~/.kubash/submodules/opendistro-build/helm/opendistro-es/values.yaml 
257d256
<       storageClass: "openebs-lvmpv"
335d333
<       storageClass: "openebs-lvmpv"
$ git branch|cat
* opendistro-1.13.1

here are the logs from the kibana container:

k logs whc01elastic-opendistro-es-kibana-78c4686fcc-lm4xf|grep -v 'disconnected before secure TLS'     
{"type":"log","@timestamp":"2021-06-27T17:41:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"telemetryManagementSection\" has been disabled since the following direct or transitive dependencies are missing or disabled: [telemetry]"}
{"type":"log","@timestamp":"2021-06-27T17:41:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"newsfeed\" is disabled."}
{"type":"log","@timestamp":"2021-06-27T17:41:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"telemetry\" is disabled."}
{"type":"log","@timestamp":"2021-06-27T17:41:21Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-06-27T17:41:21Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
{"type":"log","@timestamp":"2021-06-27T17:41:21Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
{"type":"log","@timestamp":"2021-06-27T17:41:22Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [46] plugins: [opendistroAlertingKibana,usageCollection,telemetryCollectionManager,kibanaUsageCollection,securityOss,mapsLegacy,kibanaLegacy,share,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,opendistroSecurityKibana,opendistroIndexManagementKibana,opendistroAnomalyDetectionKibana,dashboard,opendistroNotebooksKibana,visualizations,visTypeVega,visTypeTimelion,timelion,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,opendistroGanttChartKibana,visualize,opendistroReportsKibana,opendistroQueryWorkbenchKibana,esUiShared,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,bfetch]"}
{"type":"log","@timestamp":"2021-06-27T17:41:23Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-06-27T17:41:26Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from Elasticsearch nodes."}

Am I doing something wrong?

2

The last line is the one I can’t figure out:

{"type":"log","@timestamp":"2021-06-27T17:41:26Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from Elasticsearch nodes."}

but if I log into the master no problem:

➜  oltorf31 k exec -it whc01elastic-opendistro-es-master-0 -- /bin/bash
[root@whc01elastic-opendistro-es-master-0 elasticsearch]# curl -XGET https://localhost:9200 -u 'admin:admin' --insecure
{
  "name" : "whc01elastic-opendistro-es-master-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "MrEo85uVQtmcnrtPV1iQzQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

and from the kibana pod no problem:

curl -XGET https://10-244-2-79.default.pod.cluster.local:9200 -u 'admin:admin' --insecure
{
  "name" : "whc01elastic-opendistro-es-master-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "MrEo85uVQtmcnrtPV1iQzQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Maybe the kibana pod is not resolving the hostname properly?

3

Interesting. I have no experience with this in Kubernetes, but the last line you mention (Unable to retrieve version information from Elasticsearch nodes.) paired with the successful communication from the Kibana pod to ES, tells me that Kibana is either misconfigured to talk to ES or it is getting misrouted by something not reproducible via curl.

4

Ya I’m wondering what kibana is set to resolve the ES instance as?

if it is the service name:

https://whc01elastic-opendistro-es-client-service.default.svc.cluster.local

then the service seems to be acting funny:

curl -XGET https://whc01elastic-opendistro-es-client-service.default.svc.cluster.local:9200 -u 'admin:admin' --insecure
curl: (35) Encountered end of file