Docker-compose.yml in "Try Open Distro for Elasticsearch" does not work

hi,

I followed the instruction here (Download | Open Distro) and ran docker-compose but got an error after waiting for some time.

Here’s the error log. Can anyone help? Thank you.

$ docker-compose up
Creating network "open-distro-for-elasticsearch_odfe-net" with the default driver
Creating volume "open-distro-for-elasticsearch_odfe-data1" with default driver
Creating volume "open-distro-for-elasticsearch_odfe-data2" with default driver
Creating odfe-node1  ... done
Creating odfe-node2  ... done
Creating odfe-kibana ... done
Attaching to odfe-node2, odfe-kibana, odfe-node1
odfe-node2    | OpenDistro for Elasticsearch Security Demo Installer
odfe-node2    |  ** Warning: Do not use on production or public reachable systems **
odfe-node2    | Basedir: /usr/share/elasticsearch
odfe-node1    | OpenDistro for Elasticsearch Security Demo Installer
odfe-node1    |  ** Warning: Do not use on production or public reachable systems **
odfe-node2    | Elasticsearch install type: rpm/deb on CentOS Linux release 7.9.2009 (Core)
odfe-node2    | Elasticsearch config dir: /usr/share/elasticsearch/config
odfe-node2    | Elasticsearch config file: /usr/share/elasticsearch/config/elasticsearch.yml
odfe-node2    | Elasticsearch bin dir: /usr/share/elasticsearch/bin
odfe-node2    | Elasticsearch plugins dir: /usr/share/elasticsearch/plugins
odfe-node2    | Elasticsearch lib dir: /usr/share/elasticsearch/lib
odfe-node1    | Basedir: /usr/share/elasticsearch
odfe-node2    | Detected Elasticsearch Version: x-content-7.10.2
odfe-node2    | Detected Open Distro Security Version: 1.13.1.0
odfe-node2    | 
odfe-node2    | ### Success
odfe-node2    | ### Execute this script now on all your nodes and then start all nodes
odfe-node2    | ### Open Distro Security will be automatically initialized.
odfe-node2    | ### If you like to change the runtime configuration 
odfe-node2    | ### change the files in ../securityconfig and execute: 
odfe-node2    | "/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh" -cd "/usr/share/elasticsearch/plugins/opendistro_security/securityconfig" -icl -key "/usr/share/elasticsearch/config/kirk-key.pem" -cert "/usr/share/elasticsearch/config/kirk.pem" -cacert "/usr/share/elasticsearch/config/root-ca.pem" -nhnv
odfe-node2    | ### or run ./securityadmin_demo.sh
odfe-node2    | ### To use the Security Plugin ConfigurationGUI
odfe-node2    | ### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
odfe-node2    | ### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
odfe-node1    | Elasticsearch install type: rpm/deb on CentOS Linux release 7.9.2009 (Core)
odfe-node1    | Elasticsearch config dir: /usr/share/elasticsearch/config
odfe-node1    | Elasticsearch config file: /usr/share/elasticsearch/config/elasticsearch.yml
odfe-node1    | Elasticsearch bin dir: /usr/share/elasticsearch/bin
odfe-node1    | Elasticsearch plugins dir: /usr/share/elasticsearch/plugins
odfe-node1    | Elasticsearch lib dir: /usr/share/elasticsearch/lib
odfe-node1    | Detected Elasticsearch Version: x-content-7.10.2
odfe-node1    | Detected Open Distro Security Version: 1.13.1.0
odfe-node1    | 
odfe-node1    | ### Success
odfe-node1    | ### Execute this script now on all your nodes and then start all nodes
odfe-node1    | ### Open Distro Security will be automatically initialized.
odfe-node1    | ### If you like to change the runtime configuration 
odfe-node1    | ### change the files in ../securityconfig and execute: 
odfe-node1    | "/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh" -cd "/usr/share/elasticsearch/plugins/opendistro_security/securityconfig" -icl -key "/usr/share/elasticsearch/config/kirk-key.pem" -cert "/usr/share/elasticsearch/config/kirk.pem" -cacert "/usr/share/elasticsearch/config/root-ca.pem" -nhnv
odfe-node1    | ### or run ./securityadmin_demo.sh
odfe-node1    | ### To use the Security Plugin ConfigurationGUI
odfe-node1    | ### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
odfe-node1    | ### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
odfe-node2    | [2021-04-22T02:25:39,280][INFO ][o.e.n.Node               ] [odfe-node2] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.8.0-50-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
odfe-node2    | [2021-04-22T02:25:39,284][INFO ][o.e.n.Node               ] [odfe-node2] JVM home [/opt/jdk], using bundled JDK [false]
odfe-node2    | [2021-04-22T02:25:39,284][INFO ][o.e.n.Node               ] [odfe-node2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-7463953009389043866, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.policy=file:///usr/share/elasticsearch/performance-analyzer-rca/pa_config/es_security.policy, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
odfe-node1    | [2021-04-22T02:25:39,614][INFO ][o.e.n.Node               ] [odfe-node1] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.8.0-50-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
odfe-node1    | [2021-04-22T02:25:39,617][INFO ][o.e.n.Node               ] [odfe-node1] JVM home [/opt/jdk], using bundled JDK [false]
odfe-node1    | [2021-04-22T02:25:39,618][INFO ][o.e.n.Node               ] [odfe-node1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-3324437809601909963, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.policy=file:///usr/share/elasticsearch/performance-analyzer-rca/pa_config/es_security.policy, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"telemetryManagementSection\" has been disabled since the following direct or transitive dependencies are missing or disabled: [telemetry]"}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"newsfeed\" is disabled."}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"telemetry\" is disabled."}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [47] plugins: [usageCollection,telemetryCollectionManager,kibanaUsageCollection,securityOss,mapsLegacy,kibanaLegacy,opendistroAlertingKibana,share,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,opendistroGanttChartKibana,opendistroNotebooksKibana,opendistroSecurityKibana,opendistroIndexManagementKibana,opendistroAnomalyDetectionKibana,opendistroTraceAnalyticsKibana,opendistroReportsKibana,opendistroQueryWorkbenchKibana,esUiShared,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,bfetch]"}
odfe-node2    | [2021-04-22T02:25:40,549][INFO ][c.a.o.e.p.c.PluginSettings] [odfe-node2] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
odfe-node1    | [2021-04-22T02:25:40,841][INFO ][c.a.o.e.p.c.PluginSettings] [odfe-node1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.23.0.4:9200"}
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:40Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from Elasticsearch nodes."}
odfe-node2    | [2021-04-22T02:25:40,929][INFO ][c.a.o.s.s.t.OpenDistroSSLConfig] [odfe-node2] SSL dual mode is disabled
odfe-node2    | [2021-04-22T02:25:40,929][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] ES Config path is /usr/share/elasticsearch/config
odfe-node2    | [2021-04-22T02:25:41,136][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] JVM supports TLSv1.3
odfe-node2    | [2021-04-22T02:25:41,137][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively
odfe-node1    | [2021-04-22T02:25:41,205][INFO ][c.a.o.s.s.t.OpenDistroSSLConfig] [odfe-node1] SSL dual mode is disabled
odfe-node1    | [2021-04-22T02:25:41,206][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] ES Config path is /usr/share/elasticsearch/config
odfe-node1    | [2021-04-22T02:25:41,409][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] JVM supports TLSv1.3
odfe-node1    | [2021-04-22T02:25:41,410][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively
odfe-node2    | [2021-04-22T02:25:41,623][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] TLS Transport Client Provider : JDK
odfe-node2    | [2021-04-22T02:25:41,624][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] TLS Transport Server Provider : JDK
odfe-node2    | [2021-04-22T02:25:41,624][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] TLS HTTP Provider             : JDK
odfe-node2    | [2021-04-22T02:25:41,624][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
odfe-node2    | [2021-04-22T02:25:41,624][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node2] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
odfe-node2    | [2021-04-22T02:25:41,794][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] Clustername: odfe-cluster
odfe-node2    | [2021-04-22T02:25:41,797][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] Directory /usr/share/elasticsearch/config has insecure file permissions (should be 0700)
odfe-node2    | [2021-04-22T02:25:41,797][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)
odfe-node2    | [2021-04-22T02:25:41,797][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)
odfe-node2    | [2021-04-22T02:25:41,797][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/esnode-key.pem has insecure file permissions (should be 0600)
odfe-node2    | [2021-04-22T02:25:41,797][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/kirk.pem has insecure file permissions (should be 0600)
odfe-node2    | [2021-04-22T02:25:41,798][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/esnode.pem has insecure file permissions (should be 0600)
odfe-node2    | [2021-04-22T02:25:41,798][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/root-ca.pem has insecure file permissions (should be 0600)
odfe-node2    | [2021-04-22T02:25:41,798][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] File /usr/share/elasticsearch/config/kirk-key.pem has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:41,850][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] TLS Transport Client Provider : JDK
odfe-node1    | [2021-04-22T02:25:41,850][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] TLS Transport Server Provider : JDK
odfe-node1    | [2021-04-22T02:25:41,850][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] TLS HTTP Provider             : JDK
odfe-node1    | [2021-04-22T02:25:41,850][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
odfe-node1    | [2021-04-22T02:25:41,850][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [odfe-node1] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
odfe-node2    | [2021-04-22T02:25:41,982][INFO ][c.a.o.j.JobSchedulerPlugin] [odfe-node2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
odfe-node2    | [2021-04-22T02:25:41,986][INFO ][c.a.o.j.JobSchedulerPlugin] [odfe-node2] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
odfe-node2    | [2021-04-22T02:25:42,005][INFO ][c.a.o.j.JobSchedulerPlugin] [odfe-node2] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
odfe-node2    | [2021-04-22T02:25:42,007][INFO ][o.e.p.PluginsService     ] [odfe-node2] loaded module [aggs-matrix-stats]
odfe-node2    | [2021-04-22T02:25:42,011][INFO ][o.e.p.PluginsService     ] [odfe-node2] loaded plugin [opendistro_security]
odfe-node2    | [2021-04-22T02:25:42,025][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node2] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml
odfe-node2    | [2021-04-22T02:25:42,034][DEPRECATION][o.e.d.c.s.Settings       ] [odfe-node2] [node.max_local_storage_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
odfe-node2    | [2021-04-22T02:25:42,040][INFO ][o.e.e.NodeEnvironment    ] [odfe-node2] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/nvme0n1p2)]], net usable_space [186.7gb], net total_space [1.8tb], types [ext4]
odfe-node2    | [2021-04-22T02:25:42,040][INFO ][o.e.e.NodeEnvironment    ] [odfe-node2] heap size [512mb], compressed ordinary object pointers [true]
odfe-node2    | [2021-04-22T02:25:42,069][INFO ][o.e.n.Node               ] [odfe-node2] node name [odfe-node2], node ID [Z8i1NTzRSfKlJx0m7dn0yQ], cluster name [odfe-cluster], roles [master, remote_cluster_client, data, ingest]
odfe-node1    | [2021-04-22T02:25:42,075][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] Clustername: odfe-cluster
odfe-node1    | [2021-04-22T02:25:42,079][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] Directory /usr/share/elasticsearch/config has insecure file permissions (should be 0700)
odfe-node1    | [2021-04-22T02:25:42,079][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,080][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,080][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/esnode-key.pem has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,080][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/kirk.pem has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,080][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/esnode.pem has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,080][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/root-ca.pem has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,080][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] File /usr/share/elasticsearch/config/kirk-key.pem has insecure file permissions (should be 0600)
odfe-node1    | [2021-04-22T02:25:42,352][INFO ][c.a.o.j.JobSchedulerPlugin] [odfe-node1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
odfe-node1    | [2021-04-22T02:25:42,356][INFO ][c.a.o.j.JobSchedulerPlugin] [odfe-node1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
odfe-node1    | [2021-04-22T02:25:42,377][INFO ][c.a.o.j.JobSchedulerPlugin] [odfe-node1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
odfe-node1    | [2021-04-22T02:25:42,379][INFO ][o.e.p.PluginsService     ] [odfe-node1] loaded module [aggs-matrix-stats]
<REMOVED FOR BREVITY>
odfe-node1    | [2021-04-22T02:25:42,383][INFO ][o.e.p.PluginsService     ] [odfe-node1] loaded plugin [opendistro_security]
odfe-node1    | [2021-04-22T02:25:42,400][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [odfe-node1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml
odfe-node1    | [2021-04-22T02:25:42,411][DEPRECATION][o.e.d.c.s.Settings       ] [odfe-node1] [node.max_local_storage_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
odfe-node1    | [2021-04-22T02:25:42,419][INFO ][o.e.e.NodeEnvironment    ] [odfe-node1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/nvme0n1p2)]], net usable_space [186.7gb], net total_space [1.8tb], types [ext4]
odfe-node1    | [2021-04-22T02:25:42,420][INFO ][o.e.e.NodeEnvironment    ] [odfe-node1] heap size [512mb], compressed ordinary object pointers [true]
odfe-node1    | [2021-04-22T02:25:42,453][INFO ][o.e.n.Node               ] [odfe-node1] node name [odfe-node1], node ID [I6G_bXncQM2KOD6Kv9wA2Q], cluster name [odfe-cluster], roles [master, remote_cluster_client, data, ingest]
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:43Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.23.0.4:9200"}
odfe-node2    | [2021-04-22T02:25:43,998][DEPRECATION][o.e.d.c.s.Settings       ] [odfe-node2] [node.max_local_storage_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
odfe-node1    | [2021-04-22T02:25:44,498][DEPRECATION][o.e.d.c.s.Settings       ] [odfe-node1] [node.max_local_storage_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
odfe-node2    | [2021-04-22T02:25:44,797][WARN ][c.a.o.s.c.Salt           ] [odfe-node2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
odfe-node2    | [2021-04-22T02:25:44,810][INFO ][c.a.o.s.a.i.AuditLogImpl ] [odfe-node2] Message routing enabled: true
odfe-node2    | [2021-04-22T02:25:44,841][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [odfe-node2] <NONE> indices are made immutable.
odfe-node2    | [2021-04-22T02:25:45,086][INFO ][c.a.o.a.b.ADCircuitBreakerService] [odfe-node2] Registered memory breaker.
odfe-node2    | [2021-04-22T02:25:45,302][INFO ][o.e.t.NettyAllocator     ] [odfe-node2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
odfe-node2    | [2021-04-22T02:25:45,373][INFO ][o.e.d.DiscoveryModule    ] [odfe-node2] using discovery type [zen] and seed hosts providers [settings]
odfe-node1    | [2021-04-22T02:25:45,376][WARN ][c.a.o.s.c.Salt           ] [odfe-node1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
odfe-node1    | [2021-04-22T02:25:45,391][INFO ][c.a.o.s.a.i.AuditLogImpl ] [odfe-node1] Message routing enabled: true
odfe-node1    | [2021-04-22T02:25:45,422][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [odfe-node1] <NONE> indices are made immutable.
odfe-node1    | [2021-04-22T02:25:45,618][INFO ][c.a.o.a.b.ADCircuitBreakerService] [odfe-node1] Registered memory breaker.
odfe-node2    | [2021-04-22T02:25:45,689][WARN ][o.e.g.DanglingIndicesState] [odfe-node2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
odfe-node1    | [2021-04-22T02:25:45,828][INFO ][o.e.t.NettyAllocator     ] [odfe-node1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:45Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.23.0.4:9200"}
odfe-node1    | [2021-04-22T02:25:45,886][INFO ][o.e.d.DiscoveryModule    ] [odfe-node1] using discovery type [zen] and seed hosts providers [settings]
odfe-node2    | [2021-04-22T02:25:45,903][INFO ][c.a.o.e.p.h.c.PerformanceAnalyzerConfigAction] [odfe-node2] PerformanceAnalyzer Enabled: false
odfe-node2    | [2021-04-22T02:25:45,963][INFO ][o.e.n.Node               ] [odfe-node2] initialized
odfe-node2    | [2021-04-22T02:25:45,963][INFO ][o.e.n.Node               ] [odfe-node2] starting ...
odfe-node2    | [2021-04-22T02:25:46,045][INFO ][o.e.t.TransportService   ] [odfe-node2] publish_address {172.23.0.3:9300}, bound_addresses {0.0.0.0:9300}
odfe-node1    | [2021-04-22T02:25:46,118][WARN ][o.e.g.DanglingIndicesState] [odfe-node1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
odfe-node2    | [2021-04-22T02:25:46,157][INFO ][o.e.b.BootstrapChecks    ] [odfe-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks
odfe-node2    | ERROR: [1] bootstrap checks failed
odfe-node2    | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
odfe-node2    | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/odfe-cluster.log
odfe-node2    | [2021-04-22T02:25:46,163][INFO ][o.e.n.Node               ] [odfe-node2] stopping ...
odfe-node2    | [2021-04-22T02:25:46,163][INFO ][c.a.o.s.a.r.AuditMessageRouter] [odfe-node2] Closing AuditMessageRouter
odfe-node2    | [2021-04-22T02:25:46,164][INFO ][c.a.o.s.a.s.SinkProvider ] [odfe-node2] Closing InternalESSink
odfe-node2    | [2021-04-22T02:25:46,164][INFO ][c.a.o.s.a.s.SinkProvider ] [odfe-node2] Closing DebugSink
odfe-node2    | [2021-04-22T02:25:46,173][INFO ][o.e.n.Node               ] [odfe-node2] stopped
odfe-node2    | [2021-04-22T02:25:46,174][INFO ][o.e.n.Node               ] [odfe-node2] closing ...
odfe-node2    | [2021-04-22T02:25:46,180][INFO ][c.a.o.s.a.i.AuditLogImpl ] [odfe-node2] Closing AuditLogImpl
odfe-node2    | [2021-04-22T02:25:46,183][INFO ][o.e.n.Node               ] [odfe-node2] closed
odfe-node1    | [2021-04-22T02:25:46,285][INFO ][c.a.o.e.p.h.c.PerformanceAnalyzerConfigAction] [odfe-node1] PerformanceAnalyzer Enabled: false
odfe-node1    | [2021-04-22T02:25:46,341][INFO ][o.e.n.Node               ] [odfe-node1] initialized
odfe-node1    | [2021-04-22T02:25:46,341][INFO ][o.e.n.Node               ] [odfe-node1] starting ...
odfe-node1    | [2021-04-22T02:25:46,436][INFO ][o.e.t.TransportService   ] [odfe-node1] publish_address {172.23.0.4:9300}, bound_addresses {0.0.0.0:9300}
odfe-node2 exited with code 78
odfe-node1    | [2021-04-22T02:25:46,533][INFO ][o.e.b.BootstrapChecks    ] [odfe-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
odfe-node1    | ERROR: [1] bootstrap checks failed
odfe-node1    | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
odfe-node1    | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/odfe-cluster.log
odfe-node1    | [2021-04-22T02:25:46,537][INFO ][c.a.o.s.a.r.AuditMessageRouter] [odfe-node1] Closing AuditMessageRouter
odfe-node1    | [2021-04-22T02:25:46,537][INFO ][o.e.n.Node               ] [odfe-node1] stopping ...
odfe-node1    | [2021-04-22T02:25:46,538][INFO ][c.a.o.s.a.s.SinkProvider ] [odfe-node1] Closing InternalESSink
odfe-node1    | [2021-04-22T02:25:46,538][INFO ][c.a.o.s.a.s.SinkProvider ] [odfe-node1] Closing DebugSink
odfe-node1    | [2021-04-22T02:25:46,545][INFO ][o.e.n.Node               ] [odfe-node1] stopped
odfe-node1    | [2021-04-22T02:25:46,546][INFO ][o.e.n.Node               ] [odfe-node1] closing ...
odfe-node1    | [2021-04-22T02:25:46,550][INFO ][c.a.o.s.a.i.AuditLogImpl ] [odfe-node1] Closing AuditLogImpl
odfe-node1    | [2021-04-22T02:25:46,552][INFO ][o.e.n.Node               ] [odfe-node1] closed
odfe-node1 exited with code 78
odfe-kibana   | {"type":"log","@timestamp":"2021-04-22T02:25:48Z","tags":["error","elasticsearch","data"],"pid":1,"message":"[ConnectionError]: getaddrinfo ENOTFOUND odfe-node1 odfe-node1:9200"}
odfe-node2    | ERROR: [1] bootstrap checks failed
odfe-node2    | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Run this on docker host machine and restart docker service

sudo sysctl -w vm.max_map_count=262144

or add “vm.max_map_count=262144” to /etc/sysctl.conf

2 Likes

Thanks, it works now.