Describe the bug
I have installed OpenDistro plugin using the following command and created the following config files manually using this doc. I have configured the elasticsearch.yml file with the created pem files.
######## Start OpenDistro for Elasticsearch Security Configuration ########
# WARNING: revise all the lines below before you go into production
opendistro_security.ssl.transport.pemcert_filepath: node.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: node.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
- O=Test,L=Chennai, C=IN
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
opendistro_security.system_indices.enabled: true
opendistro_security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3
######## End OpenDistro for Elasticsearch Security Configuration ########
But I got the following error while starting the server,
Error:
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:700) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:642) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:328) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:691) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:642) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:328) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more
**Caused by: org.elasticsearch.ElasticsearchException: Unable to read /usr/share/elasticsearch/config/esnode.pem (/usr/share/elasticsearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath**
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:929) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:226) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:350) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:247) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.<init>(DefaultOpenDistroSecurityKeyStore.java:168) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.<init>(OpenDistroSecuritySSLPlugin.java:210) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.<init>(OpenDistroSecurityPlugin.java:244) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:691) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:642) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:328) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more
uncaught exception in thread [main]
Why it’s expecting esnode.pem instead of using node.pem?
@sivasakthi Have you checked the running container against elasticsearch.yml? Does it contain the one you’ve created or has the demo one?
It looks like demo installation could overwrite your custom elasticsearch.yml file.
Do you copy your custom elasticsearch.yml file into the image or attach it with volumes?
@pablo I don’t have a path /etc/elasticsearch in docker container. There is no file I have elasticsearch.yml only under /usr/share/elasticsearch/config directory.
---
# This is the main Open Distro Security configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
# http_enabled: true
# transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "anonymous"
# and one role named "anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
# basic (challenging)
# proxy (not challenging, needs xff)
# kerberos (challenging)
# clientcert (not challenging, needs https)
# jwt (not challenging)
# host (not challenging) #DEPRECATED, will be removed in a future version.
# host based authentication is configurable in roles_mapping
# Authc
# internal
# noop
# ldap
# Authz
# ldap
# noop
_meta:
type: "config"
config_version: 2
config:
dynamic:
# Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy
#multitenancy_enabled: true
#server_username: kibanaserver
#index: '.kibana'
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
#internalProxies: '.*' # trust all internal proxies, regex pattern
#remoteIpHeader: 'x-forwarded-for'
###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
###### and here https://tools.ietf.org/html/rfc7239
###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: "Authenticate via proxy"
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: "x-proxy-user"
roles_header: "x-proxy-roles"
authentication_backend:
type: noop
jwt_auth_domain:
description: "Authenticate via Json Web Token"
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
jwt_header: "Authorization"
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: "Authenticate via SSL client certificates"
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: "Authenticate via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: 'ou=people,dc=example,dc=com'
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: '(sAMAccountName={0})'
# Use this attribute from the user as username (if not set then DN is used)
username_attribute: null
authz:
roles_from_myldap:
description: "Authorize via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
# LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
type: ldap
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: 'ou=groups,dc=example,dc=com'
# Filter to search for roles (currently in the whole subtree beneath rolebase)
# {0} is substituted with the DN of the user
# {1} is substituted with the username
# {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
rolesearch: '(member={0})'
# Specify the name of the attribute which value should be substituted with {2} above
userroleattribute: null
# Roles as an attribute of the user entry
userrolename: disabled
#userrolename: memberOf
# The attribute in a role entry containing the name of that role, Default is "name".
# Can also be "dn" to use the full DN as rolename.
rolename: cn
# Resolve nested roles transitive (roles which are members of other roles and so on ...)
resolve_nested_roles: true
userbase: 'ou=people,dc=example,dc=com'
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: '(uid={0})'
# Skip users matching a user name, a wildcard or a regex pattern
#skip_users:
# - 'cn=Michael Jackson,ou*people,o=TEST'
# - '/\S*/'
roles_from_another_ldap:
description: "Authorize via another Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
#config goes here ...
# auth_failure_listeners:
# ip_rate_limiting:
# type: ip
# allowed_tries: 10
# time_window_seconds: 3600
# block_expiry_seconds: 600
# max_blocked_clients: 100000
# max_tracked_clients: 100000
# internal_authentication_backend_limiting:
# type: username
# authentication_backend: intern
# allowed_tries: 10
# time_window_seconds: 3600
# block_expiry_seconds: 600
# max_blocked_clients: 100000
# max_tracked_clients: 100000
@sivasakthi Based on your search and provided files I don’t see any issue with the security plugin.
The security plugin will use TLS certificates based on the elasticsearch.yml configuration as it can’t modify elasticsearch.yml.
Maybe the Elasticsearch service is started with elasticsearch.yml using default security plugin settings and then elasticsearch.yml is replaced with the custom one. That would explain why you see your custom elasticsearch.yml in docker’s image and why the Elasticsearch is running with default config.
I would suggest checking the workflow order of the docker image creation.
Also, try to restart the container (start/stop and not up/down) and check if the custom elasticsearch.yml is loaded.