Elasticsearch and SSL

While moving my ELK stack into Docker I wanted to enable SSL. It's allowed for free in 7.2.0 so I might as well.

Enabling SSL wasn't complicated, except for one piece. The certificates must be under the config directory for Elasticsearch. This is due to some security pieces that are in place.

To enable SSL without using a custom image, add the following environment variables to your docker-compose file.

  - xpack.security.http.ssl.enabled=true
  - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/secrets/certificate.key.pem
  - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/secrets/certificate.crt.pem
  - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/secrets/root.pub.pem
  - xpack.security.transport.ssl.enabled=false
  - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/secrets/certificate.key.pem
  - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/secrets/certificate.crt.pem
  - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/secrets/root.pub.pem

Now, using the secrets feature of Docker Swarm to add those certificates. I first tried using the regular Docker secret location, /run/secrets, because I didn't want to diverge from the Docker standard. After about an hour of fighting it I found one small form entry that explained why it needed to be there, got it. I modified my docker-compose.yml file to store the certificates in directory under the config directory. I used /usr/share/elasticsearch/config/secrets. Most guides used certs but they weren't for Docker. Since these are pulled from the Docker secrets I went with secrets.

I use 2 docker-compose files. One that contains the basic information that would run on any system (hopefully) and one that I use to augment that one and run in my real, production, environment.

My docker-compose.yml file is this:

version: '3.7'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    environment: 
      - "ES_JAVA_OPTS=-Xms7g -Xmx7g"
      - discovery.type=single-node
    ports:
      - 9200:9200
    networks:
      - ext
networks:
  ext:

And my docker-compose.prod.yml is this:

version: "3.7"
services:
  elasticsearch:
    environment:
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=/usr/share/elasticsearch/config/secrets/certificate.key.pem
      - xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/secrets/certificate.crt.pem
      - xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/secrets/root.pub.pem
      - xpack.security.transport.ssl.enabled=false
      - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/secrets/certificate.key.pem
      - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/secrets/certificate.crt.pem
      - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/secrets/root.pub.pem
    volumes:
      - elasticsearchdata:/usr/share/elasticsearch/data
      - elasticsearchconfig:/usr/share/elasticsearch/config
      - elasticsearchlogs:/usr/share/elasticsearch/logs
    secrets:
      - source: root.pub.pem
        target: /usr/share/elasticsearch/config/secrets/root.pub.pem
        uid: '1000'
        gid: '1000'
      - source: certificate.key.pem
        target: /usr/share/elasticsearch/config/secrets/certificate.key.pem
        uid: '1000'
        gid: '1000'
      - source: certificate.crt.pem
        target: /usr/share/elasticsearch/config/secrets/certificate.crt.pem
        uid: '1000'
        gid: '1000'
secrets:
  root.pub.pem:
    external: true
    name: certificates.example.com.ca.pub.pem
  certificate.key.pem:
    external: true
    name: certificates.wildcard.example.com.key.pem
  certificate.crt.pem:
    external: true
    name: certificates.wildcard.example.com.pub.test.pem
volumes:
  elasticsearchdata:
    driver: local
    driver_opts:
      type: nfs
      device: ":/volumes/elasticsearch/data"
      o: "addr=${NFSSERVER},vers=4,rw"
  elasticsearchconfig:
    driver: local
    driver_opts:
      type: nfs
      device: ":/volumes/elasticsearch/config"
      o: "addr=${NFSSERVER},vers=4,rw"
  elasticsearchlogs:
    driver: local
    driver_opts:
      type: nfs
      device: ":/volumes/elasticsearch/logs"
      o: "addr=${NFSSERVER},vers=4,rw"

To deploy this to my swarm I use the following command:

docker stack deploy -c docker-compose.yml -c docker-compose.prod.yml prod --with-registry-auth

One thing I was concerned about was that the secrets are stored under the config directory which I have mounted to my NFS share. I was afraid that the individual secret file would be stored on the NFS share itself and easily accessible by anybody who could see it. There was a file that was created, but it was empty, no data. Disaster averted.

On a side note, when I first started this, Java was throwing an error stating that my certificate chain was invalid. Running openssl verified that it was in fact a valid chain. Turns out, in my public key PEM file, I had the public key listed twice. I removed the second copy and it started working.