Create Your Own Docker Registry in Docker Swarm

As I build my Docker Swarm cluster, I am needing a private registry. This is so I can host my own images internally, without needing to use a cloud service.

Some assumptions:

• I will be following the same naming as used in my other Docker posts. You can see those names here: https://www.frakkingsweet.com/docker-swarm-on-debian-and-hyper-v/#Assumptions.
• You already have a trusted certificate pair that your hosts will accept. Setting this up is way beyond the scope of this post.
• Your private key does not require a password to be used.
• Your private and public keys are stored in 2 files in the PEM format.
• Your private key is ~/certs/docker.example.com.key.pem.
• Your public key is ~/certs/docker.example.com.pub.pem.
• You store your local Docker Volumes in /data/docker.
• You are remotely working with your Docker Swarm from your client. To see how to do this: https://www.frakkingsweet.com/securing-the-remote-docker-instance/. Everything can be done from the client, except for creating the directories and htpasswd file.

To keep things simple, I'm using the local file system to store the files. To keep things secure I will be using the secrets feature of Docker to store my keys.

Build and deploy a local registry

First add your private/public key pair to the secrets in Docker.

docker secret create certificates.docker.example.com.key.pem ~/certs/docker.example.com.key.pem
docker secret create certificates.docker.example.com.pub.pem ~/certs/docker.example.com.pub.pem

If all went well you should have received an ID for each secret, something like m02d5nxibpgqxelioy5g4l0pc. We don't need it so you can ignore it. If you want to verify that the secrets are added, you can run docker secret ls

Now, we need to create the docker-compose.yml for the registry. In this file we will specify everything we need to setup the registry. The REGISTRY_HTTP_SECRET should be set to something random. This is used if you are clustering the registry, or if you don't want to see the warning about it missing. I have a password generator that runs entirely in JavaScript if you want to use it to generate the key: https://apps.frakkingsweet.com/Password/. To keep things simple, I used a 16-character field with numbers and letters only.

We'll save this docker-compose.yml as ~/registry/docker-compose.yml.

version: "3.7"
services:
  registry:
    restart: always
    image: registry:2
    ports:
      - 5000:5000
    environment:
      REGISTRY_HTTP_TLS_CERTIFICATE: /run/secrets/certificate.crt
      REGISTRY_HTTP_TLS_KEY: /run/secrets/certificate.key
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_HTTP_SECRET: <<CHANGE ME TO A RANDOM VALUE>>
    volumes:
      - type: volume
        source: registry.library
        target: /var/lib/registry
        volume:
          nocopy: true
      - type: volume
        source: registry.auth
        target: /auth
    networks:
      - web
    deploy:
      placement:
        constraints: [node.role == manager]
    secrets:
      - certificate.key
      - certificate.crt
volumes:
  registry.library:
    driver: local
    driver_opts:
      type: none
      device: /data/docker/registry/library
      o: bind
  registry.auth:
    driver: local
    driver_opts:
      type: none
      device: /data/docker/registry/auth
      o: bind
networks:
  web:
secrets:
  certificate.key:
    external: true
    name: certificates.docker.example.com.key.pem
  certificate.crt:
    external: true
    name: certificates.docker.example.com.pub.pem

Now that we have the docker-compose.yml file we need to create the required directories for it. On the Swarm master run the following as root.

mkdir -p /data/docker/registry/library
mkdir -p /data/docker/registry/auth

To create the htpasswd file so we can authenticate to the registry we'll use htpasswd from the registry container on the master node. Following the same names from my previous Docker posts, we'll use a username of user1 and a password of P@ssword1.

docker run --rm --entrypoint htpasswd registry:2 -Bbn user1 P@ssword1 > /data/docker/registry/auth/htpasswd

Now that we have everything created, directories for the volumes, secrets, htpasswd for authentication and the docker-compose.yml file, we need to create the service in the Swarm and start it up. As long as your client is configured to access the Swarm remotely, the rest of this guide will be done on the client.

docker stack deploy -c ~/registry/docker-compose.yml prod

You can see the status of the service by running:

docker service ps prod_registry

Push an image to your new registry

To push an image to the new registry, you first login, then tag the image, then push it. On our client, we will create a basic image based on the ASP.Net Core Application, tag it, then push it to our private registry.

On your client system, login to the new registry:

docker login docker.example.com:5000

Create our Dockerfile with the following contents:

FROM mcr.microsoft.com/dotnet/core/samples:aspnetapp

RUN echo "Hello from the build"

Build and tag our image:

docker image build --tag docker.example.com:5000/testsite:latest .

Push our image:

docker push docker.example.com:5000/testsite:latest .

If you get an error about missing credentials, use docker login docker.example.com:5000 the re-run the push.

Now to run the service and watch it start. The docker-compose.yml file should have the following:

version: "3.7"
services:
  testsite:
    image: docker.example.com:5000/testsite:latest
    ports:
      - 8123:80

Now start the service:

docker stack deploy -c .\docker-compose.yml prod --with-registry-auth

In a browser, you should be able to view the page that is hosted by a container that was pulled from your local registry: http://docker.example.com:8123

When deploying your stack to the Swarm and your image is on your private registry, be sure to include the --with-registry-auth argument. Otherwise, whatever node will run your container will not have access to that image and  docker service ps <service name> will show an error of No such image: <image name>.  The logs on the worker node may also have entries similar to this

Jun 19 20:33:00 docker1 dockerd[52954]: time="2019-06-19T20:33:00.737298982-06:00" level=info msg="Attempting next endpoint for pull after error: Get https://docker.example.com:5000/v2/testsite/manifests/latest: no basic auth credentials"
Jun 19 20:33:00 docker1 dockerd[52954]: time="2019-06-19T20:33:00.737977081-06:00" level=error msg="pulling image failed" error="Get https://docker.example.com:5000/v2/testsite/manifests/latest: no basic auth credentials" module=node/agent/taskmanager node.id=1jqulb76r6kehkntxa8ll4b27 service.id=x853j3aa5re9ttun1bfek1ozb task.id=uv7gnvykuk4p58th7kjzc1fyx
Jun 19 20:33:00 docker1 dockerd[52954]: time="2019-06-19T20:33:00.739003880-06:00" level=error msg="fatal task error" error="No such image: docker.example.com:5000/testsite:latest" module=node/agent/taskmanager node.id=1jqulb76r6kehkntxa8ll4b27 service.id=x853j3aa5re9ttun1bfek1ozb task.id=uv7gnvykuk4p58th7kjzc1fyx
Jun 19 20:33:00 docker1 dockerd[52954]: time="2019-06-19T20:33:00.876358232-06:00" level=warning msg="failed to deactivate service binding for container prod_testsite.1.5myssmf1bhc0ubbnuaeh9gqcy" error="No such container: prod_testsite.1.5myssmf1bhc0ubbnuaeh9gqcy" module=node/agent node.id=1jqulb76r6kehkntxa8ll4b27

And that's it for setting up a local private registry, connecting to it from your local client, pushing an image, and running that image on your Swarm cluster.