As I build my Docker Swarm cluster, I am needing a private registry. This is so I can host my own images internally, without needing to use a cloud service.
- I will be following the same naming as used in my other Docker posts. You can see those names here: https://www.frakkingsweet.com/docker-swarm-on-debian-and-hyper-v/#Assumptions.
- You already have a trusted certificate pair that your hosts will accept. Setting this up is way beyond the scope of this post.
- Your private key does not require a password to be used.
- Your private and public keys are stored in 2 files in the PEM format.
- Your private key is
- Your public key is
- You store your local Docker Volumes in
- You are remotely working with your Docker Swarm from your client. To see how to do this: https://www.frakkingsweet.com/securing-the-remote-docker-instance/. Everything can be done from the client, except for creating the directories and
To keep things simple, I'm using the local file system to store the files. To keep things secure I will be using the
secrets feature of Docker to store my keys.
Build and deploy a local registry
First add your private/public key pair to the secrets in Docker.
docker secret create certificates.docker.example.com.key.pem ~/certs/docker.example.com.key.pem docker secret create certificates.docker.example.com.pub.pem ~/certs/docker.example.com.pub.pem
If all went well you should have received an ID for each secret, something like
m02d5nxibpgqxelioy5g4l0pc. We don't need it so you can ignore it. If you want to verify that the secrets are added, you can run
docker secret ls
Now, we need to create the
docker-compose.yml for the registry. In this file we will specify everything we need to setup the registry. The
We'll save this
version: "3.7" services: registry: restart: always image: registry:2 ports: - 5000:5000 environment: REGISTRY_HTTP_TLS_CERTIFICATE: /run/secrets/certificate.crt REGISTRY_HTTP_TLS_KEY: /run/secrets/certificate.key REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm REGISTRY_HTTP_SECRET: <<CHANGE ME TO A RANDOM VALUE>> volumes: - type: volume source: registry.library target: /var/lib/registry volume: nocopy: true - type: volume source: registry.auth target: /auth networks: - web deploy: placement: constraints: [node.role == manager] secrets: - certificate.key - certificate.crt volumes: registry.library: driver: local driver_opts: type: none device: /data/docker/registry/library o: bind registry.auth: driver: local driver_opts: type: none device: /data/docker/registry/auth o: bind networks: web: secrets: certificate.key: external: true name: certificates.docker.example.com.key.pem certificate.crt: external: true name: certificates.docker.example.com.pub.pem
Now that we have the
docker-compose.yml file we need to create the required directories for it. On the Swarm master run the following as root.
mkdir -p /data/docker/registry/library mkdir -p /data/docker/registry/auth
To create the
htpasswd file so we can authenticate to the registry we'll use
htpasswd from the registry container on the master node. Following the same names from my previous Docker posts, we'll use a username of
user1 and a password of
docker run --rm --entrypoint htpasswd registry:2 -Bbn user1 P@ssword1 > /data/docker/registry/auth/htpasswd
Now that we have everything created, directories for the volumes, secrets,
htpasswd for authentication and the
docker-compose.yml file, we need to create the service in the Swarm and start it up. As long as your client is configured to access the Swarm remotely, the rest of this guide will be done on the client.
docker stack deploy -c ~/registry/docker-compose.yml prod
You can see the status of the service by running:
docker service ps prod_registry
Push an image to your new registry
To push an image to the new registry, you first login, then tag the image, then push it. On our client, we will create a basic image based on the ASP.Net Core Application, tag it, then push it to our private registry.
On your client system, login to the new registry:
docker login docker.example.com:5000
Dockerfile with the following contents:
FROM mcr.microsoft.com/dotnet/core/samples:aspnetapp RUN echo "Hello from the build"
Build and tag our image:
docker image build --tag docker.example.com:5000/testsite:latest .
Push our image:
docker push docker.example.com:5000/testsite:latest .
If you get an error about missing credentials, use
docker login docker.example.com:5000 the re-run the push.
Now to run the service and watch it start. The
docker-compose.yml file should have the following:
version: "3.7" services: testsite: image: docker.example.com:5000/testsite:latest ports: - 8123:80
Now start the service:
docker stack deploy -c .\docker-compose.yml prod --with-registry-auth
In a browser, you should be able to view the page that is hosted by a container that was pulled from your local registry:
When deploying your stack to the Swarm and your image is on your private registry, be sure to include the
--with-registry-auth argument. Otherwise, whatever node will run your container will not have access to that image and
docker service ps <service name> will show an error of
No such image: <image name>. The logs on the worker node may also have entries similar to this
Jun 19 20:33:00 docker1 dockerd: time="2019-06-19T20:33:00.737298982-06:00" level=info msg="Attempting next endpoint for pull after error: Get https://docker.example.com:5000/v2/testsite/manifests/latest: no basic auth credentials" Jun 19 20:33:00 docker1 dockerd: time="2019-06-19T20:33:00.737977081-06:00" level=error msg="pulling image failed" error="Get https://docker.example.com:5000/v2/testsite/manifests/latest: no basic auth credentials" module=node/agent/taskmanager node.id=1jqulb76r6kehkntxa8ll4b27 service.id=x853j3aa5re9ttun1bfek1ozb task.id=uv7gnvykuk4p58th7kjzc1fyx Jun 19 20:33:00 docker1 dockerd: time="2019-06-19T20:33:00.739003880-06:00" level=error msg="fatal task error" error="No such image: docker.example.com:5000/testsite:latest" module=node/agent/taskmanager node.id=1jqulb76r6kehkntxa8ll4b27 service.id=x853j3aa5re9ttun1bfek1ozb task.id=uv7gnvykuk4p58th7kjzc1fyx Jun 19 20:33:00 docker1 dockerd: time="2019-06-19T20:33:00.876358232-06:00" level=warning msg="failed to deactivate service binding for container prod_testsite.1.5myssmf1bhc0ubbnuaeh9gqcy" error="No such container: prod_testsite.1.5myssmf1bhc0ubbnuaeh9gqcy" module=node/agent node.id=1jqulb76r6kehkntxa8ll4b27
And that's it for setting up a local private registry, connecting to it from your local client, pushing an image, and running that image on your Swarm cluster.