In this post, I am going to cover how to replace dhcpcd with systemd to resolve some no route to host errors.

My cluster has been having issues with networking, usually the issues would go away, but this time it took my cluster down.

The issue I was having is that pods could not communicate with the control plane or pods on any other node. I was getting a lot of no route to host errors in my pods. This would cause things like the kubernetes-dashboard to not start or randomly crash, or nginx not being able to talk to certain pods or any number of other problems.

I use the Calico CNI inside of Kubernetes. Each node could talk to each other and they could ping the pods on the same node. However, pods could not talk to pods on any other node with the same no route to host error message. Neither curl nor ping would work, just the same no route to host error message.

Further baffling me, was the following messages in my logs on all of my nodes

Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali6376b8bb8ab: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali6376b8bb8ab: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali1eafad21ac5: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali1eafad21ac5: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali0e5d8ea32b6: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali0e5d8ea32b6: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali7cbafc8827c: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali7cbafc8827c: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: calice0e46d0038: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: calice0e46d0038: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: calib4841ccaf68: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: calib4841ccaf68: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali221dca70657: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali221dca70657: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali53a6a2bcd9a: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali53a6a2bcd9a: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: calic1dc03e67d4: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: calic1dc03e67d4: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali049b4be5258: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali049b4be5258: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali2522ec4beed: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali2522ec4beed: adding route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali859fc8c2dd6: pid -2118670846 deleted route to 169.254.0.0/16
Feb 24 13:22:51 kube1w1 dhcpcd[555]: cali859fc8c2dd6: adding route to 169.254.0.0/16

Where was it getting that 169.254.0.0/16 from? A quick ip a showed that all of my cali interfaces all had a 169.254.0.0/16 address. The link local address is assigned when a dhcp server does not respond to a request. This address was being assigned by dhcpcd.

After a discussion with my brother, he suggested trying to remove dhcpcd and use systemd-network instead.

Replacing it fixed the problem. There were probably some options I could have set, like disabling the link local part of dhcpcd, but I chose the nuclear option. Mostly because I wanted to learn something new and felt like it was the better option.

The following steps need to happen:

  • Get your current, static IP.
  • Configure systemd
  • Disable dhcpcd
  • Reboot

First, let's get your interface name, IP address and subnet mask using ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:eb:3a:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.217/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fded::dea6:32ff:feeb:3a12/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86396sec preferred_lft 14396sec
    inet6 2601:681:4400:abe:dea6:32ff:feeb:3a12/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86396sec preferred_lft 14396sec
    inet6 fe80::dea6:32ff:feeb:3a12/64 scope link
       valid_lft forever preferred_lft forever

Take note of it, in the example above it is 192.168.0.217/24 and my ethernet interface is eth0.

Next, we need to create the systemd network file.

The name and location of this file will be in /etc/systemd/network and the name should be 10-example.network. The number can be anything you want, example can be anything you want, but, the filename must end with .network. The contents will look something like this.

[Match]
Name=eth0

[Network]
DHCP=no
Address=192.168.0.217/24
Gateway=192.168.0.1
DNS=192.168.0.1
Domains=example.lan

Change eth0 to be the name of your interface and the other values to ones matching your environment.

Next, we need to configure resolved so it builds the resolv.conf file for us.

Edit the file /etc/systemd/resolved.conf and add the following to the bottom

LLMNR=no
DNSStubListener=no

Now that we have the systemd unit file and resolved configured we need to enable them and the timesyncd service. If you're managing your time synchronization outside of dhcpcd then you don't need to worry about enabling timesyncd.

sudo systemctl enable systemd-networkd.service
sudo systemctl enable systemd-resolved.service
sudo systemctl enable systemd-timesyncd.service

Next, disable dhcpcd.

sudo systemctl disable dhcpcd.service

Now, time to link resolv.conf and point it to the systemd-resolved version.

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Then, reboot. If all worked correctly, you should be able to still get in to your PI when it comes back up.

Script

Since I have 9 nodes in my cluster that I needed to do this to, I opted to write a script to do it for me. You will probably need to tweak it for your environment, so do not just run it blindly. My script expects the current IP address to be in the 192.168.0.0/24 subnet, a gateway of 192.168.0.1 and a DNS server of 192.168.0.190.

The script takes 2 parameters, -h which is the DNS name of the node, example would be kube1w1.example.com and -n is the name of the node in Kubernetes. Example would be kube1w1.

It will drain the node, because it will get rebooted. SSH into the node, get the current IPv4 address (I am only dealing with static IPv4 with my cluster). It then creates the systemd network unit, followed by configuring the resolved service. Enables systemd and disables dhcpcd. Followed by rebuilding of the resolv.conf and rebooting the node. After the ping stops, press ctrl+c and it will reenable the node in Kubernetes.

The script isn't perfect. It didn't need to be, not for this. It worked, and it worked well.

#!/bin/bash

while getopts h:n: option
do
    case $option
    in
        h) HOST=${OPTARG};;
        n) HOSTNAME=${OPTARG};;
    esac
done

echo Draining Kubernetes
kubectl drain $HOSTNAME --ignore-daemonsets --delete-emptydir-data

echo Getting the current IP address
IP=`ssh $HOST ip -4 a show dev eth0 | grep inet | grep -o '192\.168\.0\.[0-9][0-9][0-9]/24' | grep -o '192\.168\.0\.[0-9]*'`
echo Configuring $HOST to use $IP

cat <<EOF | ssh $HOST sudo tee /etc/systemd/network/10-example.network
[Match]
Name=eth0

[Network]
DHCP=no
Address=$IP/24
Gateway=192.168.0.1
DNS=192.168.0.190
Domains=example.com
EOF

echo Configuring resolved
cat <<EOF | ssh $HOST sudo tee /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001
# Google:     8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844
# Quad9:      9.9.9.9 2620:fe::fe
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=yes
#LLMNR=yes
#Cache=yes
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
LLMNR=no
DNSStubListener=no
EOF

echo Enabling services
ssh $HOST sudo systemctl enable systemd-networkd.service
ssh $HOST sudo systemctl enable systemd-resolved.service
ssh $HOST sudo systemctl enable systemd-timesyncd.service

echo Disabled dhcpd
ssh $HOST sudo systemctl disable dhcpcd.service

echo Reconfiguring resolve.conf and rebooting
ssh $HOST 'sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf && sudo reboot'

ping $HOST

echo Reenabling node
kubectl uncordon $HOSTNAME

Conclusion

This was a very annoying problem that has plagued me for a while, sometimes. Rebooting the node with the broken pod would almost always fix it for a while. There were times where it was perfectly fine though. Probably because Calico was cleaning up after dhcpcd and getting rid of bogus routes which dhcpcd would then end up restoring or something.

Anyways, thanks go to William Cooke for his assistance with pointing me down a good path in configuring networking on Debian on Raspberry Pi's.

About Calico
The value of using Calico for networking and network security for workloads and hosts.
Debian -- The Universal Operating System