Elasticsearch not connecting to each other - Opendistro security not initialized

Hi Team,

Just need an urgent help. I have set up a 3 node cluster in Windows environment.

All 3 nodes were communicating as expected when opendistro.security was disabled and accessed through http. however after setting up the certificates, they are not communicating and Im getting the message “Opendistro security not initialized”. Can someone please suggest if Im missing something to run ? should I be adding any credentials ? or am I supposed to run securityadmin.sh ? not sure how to run that in windows

please help

Hi,

I tried running the batch file however I don’t see any output and ES still says “Opendistro security not initialized”

Please help, Documents dont say anything about in windows env

Hi @shaimoh

Your OS nodes could be missing security plugin configuration.

Could you share your opensearch.yml and config.yml files?

1 Like

Hi @pablo,

Thanks a lot for the quick response.

The security plugin comes along with the download and I can see them on all 3 nodes what I have configured

ODFE

Below is the config.yml and opensearch.yml

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    #kibana:
    # Kibana multitenancy
    #multitenancy_enabled: true
    #server_username: kibanaserver
    #index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap

PFB one the opensearch.yml file… The other two nodes are configured the same way.


cluster.name: nndasProd-elk

node.name: node-1

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: C:\OpenDistroForElasticsearch\data

# Path to log files:

path.logs: C:\OpenDistroForElasticsearch\logs

# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: xx.xx.xx.xx
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["xx.xx.xx.xx"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["xx.xx.xx.xx" , "xx.xx.xx.xx" , "xx.xx.xx.xx" ]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

######## Start OpenDistro for Elasticsearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
opendistro_security.ssl.transport.pemcert_filepath: C:\OpenDistroForElasticsearch\config\xx.xx.xx.xx_cert.pem
opendistro_security.ssl.transport.pemkey_filepath: C:\OpenDistroForElasticsearch\config\xx.xx.xx.xx_key_pkcs8.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: C:\OpenDistroForElasticsearch\config\root_ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: C:\OpenDistroForElasticsearch\config\xx.xx.xx.xx_cert.pem
opendistro_security.ssl.http.pemkey_filepath: C:\OpenDistroForElasticsearch\config\xx.xx.xx.xx_key_pkcs8.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: C:\OpenDistroForElasticsearch\config\root_ca.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
  - CN=xxxx,OU=Infra,O=xxxx,L=Amsterdam, C=NL

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
opendistro_security.system_indices.enabled: true
opendistro_security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
######## End OpenDistro for Elasticsearch Security Demo Configuration ########

I havent done any changes to config.yml. Please suggest if anything needs to be updated in the config.yml and if I should run the securityadmin.bat file ? thanks for your help

@shaimoh to use the security plugin you must initialize it by uploading configuration using securityadmin.sh or securityadmin.bat scripts.

There is no difference between the scripts in terms of functionality. It means that options described for securityadmin.sh will be the same for securityadmin.bat.

1 Like

Hi @pablo , thanks for your response

I get it, So we need to upload configuration in securityadmin.bat file. I tried running the command from the link provided, no luck

Any example you can provide on the command to upload and run the file pls ?

Not sure where to start and upload the configuration. Im planning to use the internal user database for authentication so I did not do any changes in config.yml file

I have my certificates and Im using the node IP to access elasticsearch. pls suggest how to configure the securityadmin.bat. thanks a lot for your help.

Wondering why im not getting any output for the command

Pls help… Thanks in advance

Tried something like this still no luck :frowning:

@shaimoh lack of output is caused by 2> nul

1 in shell means standard output (stdout)
2 in shell means standard error (stderr)

2 > nul - means all standard error output is dumped to nul device which acts like black hole
Just remove that part and you’ll see errors.

@pablo thanks again for your response

Just to confirm, so elasticsearch needs to be started on all 3 nodes and I can run the securityadmin script on any one of the node ?

I tried the below and Im getting another error

The certificates are placed correctly on the path specified. Pls help

Along with the above error, Im getting another error stating “NoNodeAvailableException[None of the configured nodes are available:”

But ES is running on all 3 nodes

@shaimoh securityadmin.bat script is using Transport port to communicate with Elasticsearch instead of 9200. The transport port is in range 9300 to 9400. Please check if that port is exposed and is not blocked by a firewall.

However, the first error is more important as the script fails to find valid certification path.
Path to certificate file and certification path are two different things.

Path to certificate is the location of the certificate file in the OS’s filesystem.
A certificate path is a presentation of a valid trust chain that a signed SSL certificate must have.

For example:

Thanks @pablo

I just gave a try with the demo certificates instead of my own certificates and I could see it worked out

C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\tools>"C:\Program Files\Java\jdk-11.0.12\bin\java" -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=OFF -cp "C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\tools\\..\..\opendistro_security-ssl\*;C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\tools\\..\deps\*;C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\tools\\..\*;C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\tools\\..\..\..\lib\*" com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin -cd C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig -icl -nhnv -cert C:\OpenDistroForElasticsearchTest\config\kirk.pem -cacert C:\OpenDistroForElasticsearchTest\config\root-ca.pem -key C:\OpenDistroForElasticsearchTest\config\kirk-key.pem -h 10.200.59.34
Open Distro Security Admin v7
Will connect to 10.200.59.34:9300 ... done
Connected as CN=kirk,OU=client,O=client,L=test,C=de
Elasticsearch Version: 7.10.2
Open Distro Security Version: 1.13.1.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: my-application
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\
Will update '_doc/config' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with C:\OpenDistroForElasticsearchTest\plugins\opendistro_security\securityconfig\audit.yml
   SUCC: Configuration for 'audit' created or updated
Done with success

But when I try with the node certificates, it doesnt work
Im suspecting if the issue is with the admin_dn part.

Im using the node certificate path instead of a separate DN certificate (eg:like kirk.pem). Can that be the issue ? Do i need to generate a admin certificate apart from the node certificate ? Bit confused with both the certificates here :frowning:

Also @pablo , a small clarification… Are we supposed to run this securityadmin.bat on all the nodes once ES is started ? or can we run on any one of the node in the cluster ?

Thanks for your help

Hi @pablo

I have tried with demo certificates and it worked so clearly there is something wrong with the admin certificate on the admin_dn part

Currently im giving the same certificate for both SSL and admin_dn !! Is that where the issue can be ? I read somewhere that we need to generate admin certificate (eg: kirk in case of demo certificate)

Any idea how to generate those from windows ?

Thanks for your help

Hi @shaimoh

opendistro_secuirty.authcz.admin_dn defines users which can execute admin tasks.

securityadmin.bat requires an admin certificate instead of a node certificate.

The security plugin’s config is uploaded once and is shared across all the nodes in the cluster. You just need to install the plugin on all remaining nodes.