Sending signals to a service in a Swarm

I found myself needing to send a signal (SIGHUP) to all running containers for a service in my swarm. It's not as straight forward as it should be.

Using the docker command you can easily send a kill command to a running container on a single node. But it's not easy in a swarm, since container names are random-ish. The name is consistently the format of <stack>_<service>_<container number>.<randomdata>. Because of that, you would need to connect to each of your docker nodes, run docker container ls, then docker kill -s <signal> <container name>. Not a simple thing when you have many nodes.

The overall idea of this solution is to do the above steps for me on every node.

My client PC is a Windows system remotely connected to my docker swarm so I will be using PowerShell to run my docker-compose.yml file with the correct parameters.

I first created a folder, in this case I named it send-signal.

I then created 2 files, a docker-compose.yml and send-signal.ps1. I'm going to use the official docker image because it already contains the docker command and I don't need to mess with it.

My docker-compose.yml file:

version: '3.7'
services:
  send-signal:
    image: docker
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    command: sh -c 'echo "Searching for ${stack}_${service} to send ${signal}"; docker container ls --format "{{ .Names }}" | grep "${stack}_${service}\.\d*\..*" > tmp.txt; while read in; do echo "Sending ${signal} to $${in}";  docker kill -s ${signal} $${in}; done < tmp.txt'
    deploy:
      mode: global
      restart_policy:
        condition: none

The breakdown of this, we mount the docker socket so the container can issue commands to the system running the container. The deploy->mode set to global it tells docker to run it on every node. With the deploy->restart_policy->condition set to none it will run once and exit. The command is kind of obnoxious because we have to do it in a one liner. Effectively, it does the docker container ls with a format of just the name of the container. It then searches that output for the names that match the regex of stack_service.containernumber.randomjunk. It then passes that to a while loop which runs the docker kill -s <signal> <container name> command.

The docker-compose.yml makes use of the environment variables on the local system, ${stack}, ${service} an ${signal} It also uses the environment variable in the container by escaping the $, that's the $${in} part of the command.

My PowerShell script, send-signal.ps1 is as follows:

param(
    [Parameter(Mandatory=$true)]
    [string]
    $stack,

    [Parameter(Mandatory=$true)]
    [string]
    $service,

    [Parameter(Mandatory=$true)]
    [string]
    $signal
)

$env:stack=$stack
$env:service=$service
$env:signal=$signal
docker stack deploy --with-registry-auth -c "./docker-compose.yml" "tasks"

Then, to send the SIGHUP signal to all the services named prometheus in the prod stack, you would run this:

.\send-signal.ps1 -stack prod -service prometheus -signal "SIGHUP"

To see the results of the command, use docker service logs tasks_send-signal. The output should look similar to:

tasks_send-signal.0.cydc60ojv3t8@docker1    | Searching for prod_prometheus to send SIGHUP
tasks_send-signal.0.cydc60ojv3t8@docker1    | Sending SIGHUP to prod_prometheus.1.fyfiflhcvnrt8lxzqqscscmw0
tasks_send-signal.0.cydc60ojv3t8@docker1    | prod_prometheus.1.fyfiflhcvnrt8lxzqqscscmw0
tasks_send-signal.0.sb03p5vom2lx@docker3    | Searching for prod_prometheus to send SIGHUP
tasks_send-signal.0.syh32kq1vs25@docker2    | Searching for prod_prometheus to send SIGHUP

Some relevant posts:

GHOST_URL/cleaning-up-your-docker-swarm-cluster/

GHOST_URL/remote-docker-instance/

GHOST_URL/securing-the-remote-docker-instance/

GHOST_URL/docker-swarm-on-debian-and-hyper-v/