Hi,
Context -
I am trying to update security configurations via REST API. These parameters have been set in elasticsearch.yml.
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
opendistro_security.unsupported.restapi.allow_securityconfig_modification=true
I am able to update configurations for roles,users etc (i.e. create/patch/delete etc) using the REST API.
To enable modification of configuration (i.e. authc/authz in config.yml), have added the parameter opendistro_security.unsupported.restapi.allow_securityconfig_modification as suggested in API - Open Distro Documentation.
I am also able to get the security plugin configurations in json using:
GET _opendistro/_security/api/securityconfig
and also update by uploading the entire json using below request followed by complete json
PUT _opendistro/_security/api/securityconfig/config
For ex.
curl -u admin:admin -k -XPUT https://odfe-opendistro-es-client-service.shiv.svc.cluster.local:9200/_opendistro/_security/api/securityconfig/config -H ‘Content-Type: application/json’ -d ’
{“dynamic”:{“filtered_alias_mode”:“warn”,“disable_rest_auth”:false,“disable_intertransport_auth”:false,“respect_request_indices_options”:false,“kibana”:{“multitenancy_enabled”:true,“server_username”:“kibanaserver”,“index”:“.kibana”},“http”:{“anonymous_auth_enabled”:false,“xff”:{“enabled”:false,“internalProxies”:“192\.168\.0\.10|192\.168\.0\.11”,“remoteIpHeader”:“X-Forwarded-For”}},“authc”:{“jwt_auth_domain”:{“http_enabled”:false,“transport_enabled”:false,“order”:0,“http_authenticator”:{“challenge”:false,“type”:“jwt”,“config”:{“signing_key”:“base64 encoded HMAC key or public RSA/ECDSA pem key”,“jwt_header”:“Authorization”}},“authentication_backend”:{“type”:“noop”,“config”:{}},“description”:“Authenticate via Json Web Token”},“openid_auth_domain”:{“http_enabled”:true,“transport_enabled”:true,“order”:1,“http_authenticator”:{“challenge”:false,“type”:“openid”,“config”:{“subject_key”:“preferred_username”,“roles_key”:“roles”,“openid_connect_url”:“https://10.10.10.1:8443/auth/realms/elk/.well-known/openid-configuration",“openid_connect_idp”:{“trust_all”:true,“enable_ssl”:true,“verify_hostnames”:false}}},“authentication_backend”:{“type”:“noop”,“config”:{}}},“ldap”:{“http_enabled”:false,“transport_enabled”:false,“order”:5,“http_authenticator”:{“challenge”:false,“type”:“basic”,“config”:{}},“authentication_backend”:{“type”:“ldap”,“config”:{“enable_ssl”:false,“enable_start_tls”:false,“enable_ssl_client_auth”:false,“verify_hostnames”:true,“hosts”:[“localhost:8389”],“userbase”:“ou=people,dc=example,dc=com”,“usersearch”:“(sAMAccountName={0})”}},“description”:"Authenticate via LDAP or Active Directory”},“basic_internal_auth_domain”:{“http_enabled”:true,“transport_enabled”:true,“order”:4,“http_authenticator”:{“challenge”:true,“type”:“basic”,“config”:{}},“authentication_backend”:{“type”:“intern”,“config”:{}},“description”:“Authenticate via HTTP Basic against internal users database”},“proxy_auth_domain”:{“http_enabled”:false,“transport_enabled”:false,“order”:3,“http_authenticator”:{“challenge”:false,“type”:“proxy”,“config”:{“user_header”:“x-proxy-user”,“roles_header”:“x-proxy-roles”}},“authentication_backend”:{“type”:“noop”,“config”:{}},“description”:“Authenticate via proxy”},“clientcert_auth_domain”:{“http_enabled”:false,“transport_enabled”:false,“order”:2,“http_authenticator”:{“challenge”:false,“type”:“clientcert”,“config”:{“username_attribute”:“cn”}},“authentication_backend”:{“type”:“noop”,“config”:{}},“description”:“Authenticate via SSL client certificates”},“kerberos_auth_domain”:{“http_enabled”:false,“transport_enabled”:false,“order”:6,“http_authenticator”:{“challenge”:true,“type”:“kerberos”,“config”:{“krb_debug”:false,“strip_realm_from_principal”:true}},“authentication_backend”:{“type”:“noop”,“config”:{}}}},“authz”:{“roles_from_another_ldap”:{“http_enabled”:false,“transport_enabled”:false,“authorization_backend”:{“type”:“ldap”,“config”:{}},“description”:“Authorize via another Active Directory”},“roles_from_myldap”:{“http_enabled”:false,“transport_enabled”:false,“authorization_backend”:{“type”:“ldap”,“config”:{“enable_ssl”:false,“enable_start_tls”:false,“enable_ssl_client_auth”:false,“verify_hostnames”:true,“hosts”:[“localhost:8389”],“rolebase”:“ou=groups,dc=example,dc=com”,“rolesearch”:“(member={0})”,“userrolename”:“disabled”,“rolename”:“cn”,“resolve_nested_roles”:true,“userbase”:“ou=people,dc=example,dc=com”,“usersearch”:“(uid={0})”}},“description”:“Authorize via LDAP or Active Directory”}},“auth_failure_listeners”:{},“do_not_fail_on_forbidden”:false,“multi_rolespan_enabled”:true,“hosts_resolver_mode”:“ip-only”,“do_not_fail_on_forbidden_empty”:false}}’
Issue -
Issue is seen while trying to PATCH.
Tried to run patch to update just one parameter from the security config - intention is to just update 1-2 individual attributes of existing configuration and not upload the entire json document for the same -
Using reference from API - Open Distro Documentation, the following cmd was tried -
curl -u admin:admin -k -XPATCH https://odfe-opendistro-es-client-service.shiv.svc.cluster.local:9200/_opendistro/_security/api/securityconfig -H ‘Content-Type: application/json’ -d ’
[
{
“op”: “replace”, “path”: “/config/dynamic/authc/openid_auth_domain/transport_enabled”, “value”: “false”
}
]
’
It resulted in {“status”:“OK”,“message”:“Resource updated.”}
But after that, the cluster is not accessible. Rest api gives empty response for all calls.
$ curl -k -XGET https://odfe-opendistro-es-client-service.shiv-sg.svc.cluster.local:9200/ -uadmin:admin $ curl -k -XGET https://odfe-opendistro-es-client-service.shiv-sg.svc.cluster.local:9200/_cluster/health -uadmin:admin
And these errors are seen in elasticsearch-client pod logs -
[2020-11-06T10:15:43,491][ERROR][c.a.d.a.h.j.HTTPJwtAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] Error creating JWT authenticator: io.jsonwebtoken.io.DecodingException: Illegal base64 character: ’ '. JWT authentication will not work
io.jsonwebtoken.io.DecodingException: Illegal base64 character: ’ ’
at io.jsonwebtoken.io.Base64.ctoi(Base64.java:206) ~[jjwt-api-0.10.5.jar:?]
at io.jsonwebtoken.io.Base64.decodeFast(Base64.java:255) ~[jjwt-api-0.10.5.jar:?]
at io.jsonwebtoken.io.Base64Decoder.decode(Base64Decoder.java:21) ~[jjwt-api-0.10.5.jar:?]
at io.jsonwebtoken.io.Base64Decoder.decode(Base64Decoder.java:8) ~[jjwt-api-0.10.5.jar:?]
at io.jsonwebtoken.io.ExceptionPropagatingDecoder.decode(ExceptionPropagatingDecoder.java:21) ~[jjwt-api-0.10.5.jar:?]
at com.amazon.dlic.auth.http.jwt.HTTPJwtAuthenticator.(HTTPJwtAuthenticator.java:81) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:233) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:334) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:251) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigModelV7.(DynamicConfigModelV7.java:60) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:198) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:306) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:295) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:278) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:116) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:59) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.8.0.jar:7.8.0]
at com.amazon.opendistro.elasticsearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:48) [opendistro_performance_analyzer-1.9.0.1.jar:1.9.0.1]
at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:182) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:164) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:92) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:613) [opendistro_security-1.9.0.0.jar:1.9.0.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:801) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-11-06T10:15:43,520][ERROR][c.a.d.a.h.k.HTTPSpnegoAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] acceptor_principal must not be null or empty. Kerberos authentication will not work
[2020-11-06T10:15:43,520][ERROR][c.a.d.a.h.k.HTTPSpnegoAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] opendistro_security.kerberos.acceptor_keytab_filepath must not be null or empty. Kerberos authentication will not work
[2020-11-06T10:15:43,540][INFO ][stdout ] [odfe-opendistro-es-client-546b5fc75c-5mxjx] [FINE] No subscribers registered for event class com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2020-11-06T10:15:43,544][INFO ][stdout ] [odfe-opendistro-es-client-546b5fc75c-5mxjx] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2020-11-06T10:15:44,469][ERROR][c.a.d.a.h.j.HTTPJwtAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] Missing Signing Key. JWT authentication will not work
[2020-11-06T10:15:46,970][ERROR][c.a.d.a.h.j.HTTPJwtAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] Missing Signing Key. JWT authentication will not work
[2020-11-06T10:15:47,956][ERROR][c.a.d.a.h.j.HTTPJwtAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] Missing Signing Key. JWT authentication will not work
[2020-11-06T10:15:49,472][ERROR][c.a.d.a.h.j.HTTPJwtAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] Missing Signing Key. JWT authentication will not work
[2020-11-06T10:15:51,973][ERROR][c.a.d.a.h.j.HTTPJwtAuthenticator] [odfe-opendistro-es-client-546b5fc75c-5mxjx] Missing Signing Key. JWT authentication will not work
As you can see, only openid_auth_domain and basic_auth have been enabled, i havent even enabled jwt_auth_domain but I get exceptions related to that.
What could be the issue here? Is the command to patch security configurations as mentioned in the docs not right?