Rancher, Kubernetes and Debian 10 woes

I got my Docker server automation scripts running a few days ago and started playing with Kubernetes and Rancher. It didn't work. At all.

I don't know if it's environment specific or not, but on my fresh, brand new installs, of Debian 10, Kubernetes pods could not talk to the outside world, nor could they talk to each other on the same node or different. Obviously this doesn't bode well for clusters because they need to talk to each other.

Because it was having problems with containers talking to each other on the same node I was thankfully able to rule out things like subnet collisions and other network topology issues like BGB routing, network firewalls, proxies, etc.

This was a very strange issue. About half of my nodes had the problem and the other half didn't. I remind you, these were fresh, brand new, installs of the OS. Exact same steps on all nodes. The installs were automated so no differences during the install. I could re-install a bad node and it would work. But the cluster was dead so I started over, then another node didn't, completely random. Another interesting thing is that containers started up using the docker CLI worked fine. It was only the Kubernetes pods that were having problems. That's how strange this issue is.

After tons of digging through logs and trying different things I found that the FORWARD policy in iptables was set to DROP.  Note: Docker does this by design when it starts. I changed that default policy to ACCEPT with sudo iptables -P FORWARD ACCEPT and all of the sudden my containers started talking and things things were looking much better. I then restarted my servers just to see if it would stick, it didn't. Set up the iptables auto save/restore. It didn't work, it applied the rules before docker assigned the policy to drop the packets. I needed to assign the ACCEPT policy after the Docker service started.

To make sure this doesn't start breaking on my nodes that don't have it, I put this fix on them as well.

What I did was create a new systemd service/unit, set the requires to docker.service and the command to start a shell, wait 20 seconds and issue the iptables command.

Here's my service, the file is saved at /etc/systemd/system/forwardaccept.service

[Unit]
After=docker.service
Description=Set firewall forward policy to accept

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'sleep 20; /usr/sbin/iptables -P FORWARD ACCEPT'

[Install]
WantedBy=multi-user.target

It's pretty basic, and it works. To start it, use

sudo systemctl daemon-reload
sudo systemctl enable forwardaccept.service

This command gets forked off to the background, so after about 20 seconds from your server booting up you should see that the FORWARD policy changes from DROP to ACCEPT.

While the policy was set to DROP I was unable to install Kubernetes. I was getting all sorts of random errors. Particularly around not being able to validate the control plane and etcd containers. Even after changing it to ACCEPT I had to rebuild the VM and start from scratch. Installing Kubernetes puts a bunch of stuff in the old "legacy" iptables chains. And there's not much of a cleanup of a failed install. Since I automated the install of a full Debian system with Docker and it only takes 3 minutes, I didn't really loose much time.

There were some posts out there about changing your iptables command using update-alternatives to the legacy one. Don't do that. It doesn't work on newer versions of Kubernetes. They fixed the issue that needed that particular work around.

Conclusion

This was probably one of the toughest issues I've had to troubleshoot in the recent past. I'm glad I found a work around for it, but I wish Docker didn't set your firewall to drop all of the forwarded packets. I suspect there's some issue in the bridge/iptables/nftables stuff in the newer kernels that is causing the problems.

One good thing that came out of this is that I learned how to start a script at startup using systemd. So that's cool.

Hopefully this post will help some other poor soul that has run across this particularly obnoxious issue.