Sending signals to a service in a Swarm
I found myself needing to send a signal (SIGHUP) to all running containers for a service in my swarm. It's not as straight forward as it should be.
I found myself needing to send a signal (SIGHUP) to all running containers for a service in my swarm. It's not as straight forward as it should be.
Using the docker
command you can easily send a kill
command to a running container on a single node. But it's not easy in a swarm, since container names are random-ish. The name is consistently the format of <stack>_<service>_<container number>.<randomdata>
. Because of that, you would need to connect to each of your docker nodes, run docker container ls
, then docker kill -s <signal> <container name>
. Not a simple thing when you have many nodes.
The overall idea of this solution is to do the above steps for me on every node.
My client PC is a Windows system remotely connected to my docker swarm so I will be using PowerShell to run my docker-compose.yml
file with the correct parameters.
I first created a folder, in this case I named it send-signal
.
I then created 2 files, a docker-compose.yml
and send-signal.ps1
. I'm going to use the official docker
image because it already contains the docker command and I don't need to mess with it.
My docker-compose.yml
file:
version: '3.7'
services:
send-signal:
image: docker
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
command: sh -c 'echo "Searching for ${stack}_${service} to send ${signal}"; docker container ls --format "{{ .Names }}" | grep "${stack}_${service}\.\d*\..*" > tmp.txt; while read in; do echo "Sending ${signal} to $${in}"; docker kill -s ${signal} $${in}; done < tmp.txt'
deploy:
mode: global
restart_policy:
condition: none
The breakdown of this, we mount the docker socket so the container can issue commands to the system running the container. The deploy->mode
set to global
it tells docker to run it on every node. With the deploy->restart_policy->condition
set to none
it will run once and exit. The command
is kind of obnoxious because we have to do it in a one liner. Effectively, it does the docker container ls
with a format of just the name of the container. It then searches that output for the names that match the regex of stack_service.containernumber.randomjunk
. It then passes that to a while loop which runs the docker kill -s <signal> <container name>
command.
The docker-compose.yml
makes use of the environment variables on the local system, ${stack}
, ${service}
an ${signal}
It also uses the environment variable in the container by escaping the $
, that's the $${in}
part of the command.
My PowerShell script, send-signal.ps1
is as follows:
param(
[Parameter(Mandatory=$true)]
[string]
$stack,
[Parameter(Mandatory=$true)]
[string]
$service,
[Parameter(Mandatory=$true)]
[string]
$signal
)
$env:stack=$stack
$env:service=$service
$env:signal=$signal
docker stack deploy --with-registry-auth -c "./docker-compose.yml" "tasks"
Then, to send the SIGHUP
signal to all the services named prometheus
in the prod
stack, you would run this:
.\send-signal.ps1 -stack prod -service prometheus -signal "SIGHUP"
To see the results of the command, use docker service logs tasks_send-signal
. The output should look similar to:
tasks_send-signal.0.cydc60ojv3t8@docker1 | Searching for prod_prometheus to send SIGHUP
tasks_send-signal.0.cydc60ojv3t8@docker1 | Sending SIGHUP to prod_prometheus.1.fyfiflhcvnrt8lxzqqscscmw0
tasks_send-signal.0.cydc60ojv3t8@docker1 | prod_prometheus.1.fyfiflhcvnrt8lxzqqscscmw0
tasks_send-signal.0.sb03p5vom2lx@docker3 | Searching for prod_prometheus to send SIGHUP
tasks_send-signal.0.syh32kq1vs25@docker2 | Searching for prod_prometheus to send SIGHUP
Some relevant posts:
GHOST_URL/cleaning-up-your-docker-swarm-cluster/
GHOST_URL/remote-docker-instance/
GHOST_URL/securing-the-remote-docker-instance/
GHOST_URL/docker-swarm-on-debian-and-hyper-v/