TLS certificates and nodes_dn exceptions

I’m trying to form an opensearch app on kubernetes and having trouble setting up the certificates.
I’m getting this error on startup (which doesnt fail the pod itself):

The certificates were generated using the bash supplied here:
https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/#sample-script

This means that one node is trying to connect to another with 
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone 
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]

if I’m making requests from inside the pod it works. but from outside it’s not.
when I make a request from another pod ( I have a service setup)
I get the following error:

io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: <value>

my opensearch.yml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: opensearch-config
data:
  opensearch.yml: |
    cluster.name: opensearch-cluster

    network.host: "0.0.0.0"

    transport.host: "0.0.0.0"
    transport.tcp.port: 9300

    http.host: "0.0.0.0"
    http.port: 9200


    action.auto_create_index: true
    bootstrap.memory_lock: true


    plugins.security.authcz.admin_dn:
      - 'subject= CN=ADMIN,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
    plugins.security.nodes_dn:
      - 'subject= CN=*.opensearch-master-headless.search.svc.cluster.local,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
      - 'subject= CN=*.opensearch-master-headless.search.svc.cluster.local,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
      - 'subject= CN=*.opensearch-master-headless.search.svc.cluster.local,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'

    plugins.security.allow_unsafe_democertificates: true
    plugins.security.ssl.http.enabled: "true"

    plugins.security.ssl.http.pemtrustedcas_filepath: {{ .Values.rootCertPathOnMachine }}
    plugins.security.ssl.http.pemcert_filepath: {{ .Values.certPathOnMachine }}
    plugins.security.ssl.http.pemkey_filepath: {{ .Values.privateKeyPathOnMachine }}

    plugins.security.ssl.transport.enabled: "true"
    plugins.security.ssl.transport.enforce_hostname_verification: false
    plugins.security.ssl.transport.resolve_hostname: false

    plugins.security.ssl.transport.pemtrustedcas_filepath: {{ .Values.rootCertPathOnMachine }}
    plugins.security.ssl.transport.pemcert_filepath: {{ .Values.certPathOnMachine }}
    plugins.security.ssl.transport.pemkey_filepath: {{ .Values.privateKeyPathOnMachine }}


    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true

    plugins.security.allow_default_init_securityindex: true
    plugins.security.restapi.roles_enabled:
      ["all_access", "security_rest_api_access"]
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices:
      [
        ".opendistro-alerting-config",
        ".opendistro-alerting-alert*",
        ".opendistro-anomaly-results*",
        ".opendistro-anomaly-detector*",
        ".opendistro-anomaly-checkpoints",
        ".opendistro-anomaly-detection-state",
        ".opendistro-reports-*",
        ".opendistro-notifications-*",
        ".opendistro-notebooks",
        ".opendistro-asynchronous-search-response*",
        ".replication-metadata-store",
      ]

my deployment:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ .Values.global.name }}
  namespace: {{ .Values.global.namespace }}
  clusterName: {{ .Values.global.clusterName }}
  labels:
    app: {{ .Values.global.name }}
  annotations:
    majorVersion: "{{ include "opensearch.majorVersion" . }}"
spec:
  serviceName: "{{ .Values.global.name }}-master-headless"
  selector:
    matchLabels:
      app: {{ .Values.global.name }}
  replicas: {{  .Values.replicas }}
  template:
    metadata:
      name: {{ .Values.global.name }}
      labels:
        app: {{ .Values.global.name }}
        role: master
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox:latest
          command:
            - sysctl
            - -w
            - vm.max_map_count=262144
          securityContext:
            privileged: true
      containers:
        - name: "{{.Values.global.name }}-master"
          image: opensearchproject/opensearch:1.2.3
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
          ports:
            - containerPort: 9200
              name: rest
              protocol: TCP
            - containerPort: 9300
              name: inter-node
              protocol: TCP
          env:
            - name: node.name
              value: "{{ .Values.global.name }}"
            - name: node.master
              value: "true"
            - name: cluster.initial_master_nodes
              value: "{{ .Values.global.name }}"
            - name: discovery.seed_hosts
              value: "{{ .Values.global.name  }}"
            - name: cluster.name
              value: "{{ .Values.global.clusterName }}"
            - name: ES_JAVA_OPTS
              value: "-Xms512m -Xmx512m"
          volumeMounts:
            - name: {{ .Values.global.name }}
              mountPath: /etc/opensearch/data
            - name: config
              mountPath: /usr/share/opensearch/config/opensearch.yml
              subPath: opensearch.yml
            - name: node-key
              mountPath: {{ .Values.privateKeyPathOnMachine }}
              subPath: node-key.pem
              readOnly: true
            - name: node
              mountPath: {{ .Values.certPathOnMachine }}
              subPath: node.pem
              readOnly: true
            - name: root-ca
              mountPath: {{ .Values.rootCertPathOnMachine }}
              subPath: root-ca.pem
      volumes:
        - name: config
          configMap:
            name: opensearch-config
        - name: node
          secret:
            secretName: node
            items:
              - key: node.pem
                path: node.pem
        - name: node-key
          secret:
            secretName: node-key
            items:
              - key: node-key.pem
                path: node-key.pem
        - name: root-ca
          secret:
            secretName: root-ca
            items:
              - key: root-ca.pem
                path: root-ca.pem
  volumeClaimTemplates:
    - metadata:
        name: {{ .Values.global.name }}
        labels:
          app: {{ .Values.global.name }}
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gi

certificates seems to load ok from secrets.

@yovel Could you share requests your requests?

@pablo I was able to move