GoBGP with health checks
While rebuilding my network, I'm going through the exercise of making everything highly available with GoBGP.
While rebuilding my network, I'm going through the exercise of making everything highly available with GoBGP.
Purpose
I am currently in the process of rebuilding my entire network, moving away from my dying Raspberry Pi 4's and back to a dedicated PC. My new setup is using Proxmox for the hypervisor (or orchestrator of the hypervisor?) and micro-segmenting each component of my network, DNS/DHCP/package repository/image registry/Kubernetes clusters/etc. I am also making everything redundant.
The way I am handling the redundancy is by using BGP to distribute the traffic between each node. If the BGP neighbor stops announcing (machine is dead), then it stops sending traffic there. If the health check fails, then GoBGP will denounce itself and be removed from the peering.
My standard setup is each virtual machine has a single virtual network card. The IP announced through BGP is assigned to the loopback interface to avoid polluting the subnet with bogus IP addresses. It also allows the services to listen on it whether the network interface is up or down while the machine is booting. I then have a Docker image running GoBGP and a set of scripts. These scripts call the GoBGP CLI to manage the daemon and GoBGP neighbors.
You can see my Docker image repository here:
EdwardCookeCurrently I have 2 built-in health checks. One for doing a simple curl command and another for doing a dig dns lookup. They are all configurable and usable through command line arguments.
I added additional flexibility by making each piece of that puzzle in the image configurable and replaceable as needed via command line arguments.
As happy as I am with this setup, there is one downside to the GoBGP CLI approach, the CLI does not support setting passwords for the neighbor. You can see in this issue that I opened up, no traction yet from upstream on whether they would accept a PR or not.
osrgIf you have any questions about how I got GoBGP to do what I needed let me know in the comments, I'll be happy to help in any way I can.
Conclusion
I originally started out using ExaBGP, but that proved to be a fruitless endeavor. No matter what I did, after a few hours the daemon would stop responding and would not do anything. The process was not terminated, it just stopped doing anything. No logs, no health checks, nothing. I spent a couple of weeks troubleshooting it before finally giving up and went a different direction. I settled on GoBGP, it took a little more work to get setup, but not by much. I still had to build custom scripts to use it for what I needed. Just more of them to handle the lifecycle of the health checks.
On a side note, everything in my network is being accomplished using Terraform and cloud init using images provided by Debian with a couple of customizations. Currently all services are inside of a Docker container which is configured using Docker compose. Those docker-compose.yml files are checked into a Git repository. If there is any configuration or data storage necessary for that container it is mounted from my NAS which is encrypted and backed up to Azure.
Being able to delete a VM after I do something silly, and type terraform apply and have it rebuilt from scratch is really nice.
I am also taking this time to re-architect and re-engineer all aspects of my network. I rebuilt DNS using a multi-tier approach, another post on that is coming and moving from ISC-DHCP-Server to Kea. I'm also changing from the Docker self-hosted registry to using Harbor.
This is a big undertaking, and the underpinnings of all of it has to be stable and reliable. And GoBGP seems to fit this need nicely.
Links
EdwardCookeosrg