Frakkingsweet

GoBGP with health checks

Edward Cooke — Thu, 12 Oct 2023 01:10:23 GMT

While rebuilding my network, I'm going through the exercise of making everything highly available with GoBGP.

Purpose

I am currently in the process of rebuilding my entire network, moving away from my dying Raspberry Pi 4's and back to a dedicated PC. My new setup is using Proxmox for the hypervisor (or orchestrator of the hypervisor?) and micro-segmenting each component of my network, DNS/DHCP/package repository/image registry/Kubernetes clusters/etc. I am also making everything redundant.

The way I am handling the redundancy is by using BGP to distribute the traffic between each node. If the BGP neighbor stops announcing (machine is dead), then it stops sending traffic there. If the health check fails, then GoBGP will denounce itself and be removed from the peering.

My standard setup is each virtual machine has a single virtual network card. The IP announced through BGP is assigned to the loopback interface to avoid polluting the subnet with bogus IP addresses. It also allows the services to listen on it whether the network interface is up or down while the machine is booting. I then have a Docker image running GoBGP and a set of scripts. These scripts call the GoBGP CLI to manage the daemon and GoBGP neighbors.

You can see my Docker image repository here:

GitHub - EdwardCooke/gobgp-healthcheck

Contribute to EdwardCooke/gobgp-healthcheck development by creating an account on GitHub.

GitHubEdwardCooke

Currently I have 2 built-in health checks. One for doing a simple curl command and another for doing a dig dns lookup. They are all configurable and usable through command line arguments.

I added additional flexibility by making each piece of that puzzle in the image configurable and replaceable as needed via command line arguments.

As happy as I am with this setup, there is one downside to the GoBGP CLI approach, the CLI does not support setting passwords for the neighbor. You can see in this issue that I opened up, no traction yet from upstream on whether they would accept a PR or not.

Set neighbor password through CLI · Issue #2711 · osrg/gobgp

I’m wondering if I can set the password to talk to the neighbor through the cli when adding it. It looks like its in the API, but I can’t seem to set it, and looking at the code for the CLI it does…

GitHubosrg

If you have any questions about how I got GoBGP to do what I needed let me know in the comments, I'll be happy to help in any way I can.

Conclusion

I originally started out using ExaBGP, but that proved to be a fruitless endeavor. No matter what I did, after a few hours the daemon would stop responding and would not do anything. The process was not terminated, it just stopped doing anything. No logs, no health checks, nothing. I spent a couple of weeks troubleshooting it before finally giving up and went a different direction. I settled on GoBGP, it took a little more work to get setup, but not by much. I still had to build custom scripts to use it for what I needed. Just more of them to handle the lifecycle of the health checks.

On a side note, everything in my network is being accomplished using Terraform and cloud init using images provided by Debian with a couple of customizations. Currently all services are inside of a Docker container which is configured using Docker compose. Those docker-compose.yml files are checked into a Git repository. If there is any configuration or data storage necessary for that container it is mounted from my NAS which is encrypted and backed up to Azure.

Being able to delete a VM after I do something silly, and type terraform apply and have it rebuilt from scratch is really nice.

I am also taking this time to re-architect and re-engineer all aspects of my network. I rebuilt DNS using a multi-tier approach, another post on that is coming and moving from ISC-DHCP-Server to Kea. I'm also changing from the Docker self-hosted registry to using Harbor.

This is a big undertaking, and the underpinnings of all of it has to be stable and reliable. And GoBGP seems to fit this need nicely.

Links

GitHub - EdwardCooke/gobgp-healthcheck

Contribute to EdwardCooke/gobgp-healthcheck development by creating an account on GitHub.

GitHubEdwardCooke

GitHub - osrg/gobgp: BGP implemented in the Go Programming Language

BGP implemented in the Go Programming Language. Contribute to osrg/gobgp development by creating an account on GitHub.

GitHubosrg

Pre-pull images in the docker-in-docker image

Edward Cooke — Wed, 02 Aug 2023 02:52:27 GMT

We needed to pre-pull images into the upstream docker:dind image. Here's a way of doing this.

Overview

Our builds always use the same upstream images, we're tired of re-pulling them every time it builds. It adds a lot of unnecessary time when they can be pre-cached in our build image.

As with most things with shell scripts and Docker, you will want to do this in a Linux or WSL environment.

Solution

We already use the docker:dind image in our on-prem build environment so it made sense to continue using that image and extend it. Everywhere we looked, people say this can't be done, I didn't believe them, so I did it.

I broke down the steps, start the docker daemon, pull the images, call it a day. It wasn't as simple as that for some reasons explained below.

Problems

The Docker daemon requires --privileged on the run command. Meaning it has full access to the system. This is not supported (out of the box) by Docker.
When I finally got it running it was using the vfs volume driver, when running as a regular container it was using overlay2.
The overlay2 storage driver is not compatible with storage provided by the build context.

How I did it

Create the `buildx` build context

First step, start the docker daemon during the build. This required running the build with elevated privileges. To do this we had to create a special buildx build context with a couple of flags enabled. To create that I used the following command:

docker buildx create --name builder \
    --driver docker-container \
    --buildkitd-flags '--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host'

This command creates a new buildx build context named builder using the docker-container driver with the ability to run a privileged RUN command in the Dockerfile.

Create the `Dockerfile`

The Dockerfile is a simple one, we set the syntax to a special one that allows us to use --security=insecure in the RUN command.

# syntax=docker/dockerfile:1-labs

FROM docker:dind

COPY install.sh /opt/install.sh
RUN --security=insecure /opt/install.sh

This will run the /opt/install.sh command with elevated privileges.

Create the `install.sh` script

Here's the link to install.sh in my git repository. It's large enough to not want to paste it into my page.

prepull-dind/install.sh at main · EdwardCooke/prepull-dind

Contribute to EdwardCooke/prepull-dind development by creating an account on GitHub.

GitHubEdwardCooke

This script does the following

Creates a temporary ext4 volume and mounts it on /var/lib/docker
Starts the Docker daemon in the background
Pulls the image(s)
Stops the daemon
Moves the files from /var/lib/docker to /opt/temp
Unmounts and removes the temporary volume and file
Moves the files back

To increase the size of the temporary volume, in case you're copying in a bunch of base images, modify the dd line in install.sh to be an appropriate size. The size of this file doesn't affect your overall image since it's removed at the end.

To pull other images you will modify this file, and change the line that currently pulls alpine:latest to pull whatever images you would like.

Don't forget to mark this file as executable.

chmod +x install.sh

Build the image

Now we have all of the files in place. To build the image we will use buildx, our build context builder, allowing security.insecure and pushing into our main Docker context.

docker buildx build \
    --allow security.insecure \
    --output type=docker \
    -t mydind \
    --builder builder \
    --progress=plain \
    .

Test

To test this, we will start the docker image we just built and in another terminal, run docker image ls in that container and see the alpine:latest image is there.

Start it

docker run -it --rm --privileged --name dind mydind

Validate

docker exec -i dind docker image ls

You should see this output

REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
alpine       latest    c1aabb73d233   6 weeks ago   7.33MB

Caveat

If you want to create a volume for your /var/lib/docker directory you'll need to do a little more work. In the install.sh you won't want to move the files back to /var/lib/docker. Instead, you will need a new entrypoint script that checks to see if /var/lib/docker has anything in it and if not, recursively copy the /opt/temp over to that directory. After that, have the script execute exec dockerd-entrypoint.sh $@.

Conclusion

I had a lot of fun with this. I like little projects where everyone says you can't do something. It makes it a fun challenge.

Links

GitHub - EdwardCooke/prepull-dind

Contribute to EdwardCooke/prepull-dind development by creating an account on GitHub.

GitHubEdwardCooke

Dockerfile reference

Find all the available commands you can use in a Dockerfile and learn how to use them, including COPY, ARG, ENTRYPOINT, and more.

Docker Documentation

Docker

Cloud-init, Hyper-V and a custom image

Edward Cooke — Sun, 09 Jul 2023 17:36:17 GMT

Today I'm going to go over using Cloud-init with Hyper-V and customizing the base image.

Overview

I build a lot of VM's and tear them down throughout a general work week, I am tired of spending a bunch of time baby-sitting an installer to enter the same information all the time. I wanted to make this a single script that I can run to create an entire environment.

To accomplish this I needed an image that has Cloud-init setup in it that works with Hyper-V. I chose the Ubuntu cloud image for Azure. Azure runs on Hyper-V so it fit well right out of the box, mostly. I do need to pull out some of the Azure specific customizations. I'll cover that as well.

To customize the image and to remove the Azure components you'll need to be an Administrator on your computer. You will need Windows Subsystem for Linux installed from the Microsoft Store. The reason you need WSL from the store is that we mount the Ubuntu cloud image inside of it so we can customize it. That feature is only in the Store instance of WSL.

To create the virtual machine you will need the Windows Assessment and Deployment Kit with the Deployment Tools feature. This is used to create the iso image that contains the Cloud-init configuration.

Download and install the Windows ADK

Instructions on how to download and install the Windows ADK

Microsoft Learnwindows-driver-content

Assumptions with this post

You'll be using the Ubuntu Mantic distribution.
You have admin rights on your computer
You have Windows Subsystem for Linux already setup and running with default settings and a distribution with chroot.
You're using a directory named c:\Cloud-init to store everything.
You have the Windows Assessment and Deployment Kit with a minimum, the Deployment Tools feature installed at the default location of C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit.
You have this repository checked out

GitHub - EdwardCooke/hyperv-cloud-init: Easy scripts for creating a cloud-init based VM in Hyper-V

Easy scripts for creating a cloud-init based VM in Hyper-V - GitHub - EdwardCooke/hyperv-cloud-init: Easy scripts for creating a cloud-init based VM in Hyper-V

GitHubEdwardCooke

Create the base image

Download

First step in creating the base image, download the Azure Cloud-init image from Ubuntu's cloud image site linked here

Ubuntu Cloud Images - the official Ubuntu images for public clouds, Openstack, KVM and LXD

Ubuntu Cloud Images are the official Ubuntu images that have been customized by Canonical to run on public clouds that provide Ubuntu Certified Images, Openstack, KVM, LXD and more.

the official Ubuntu images for public clouds, Openstack, KVM and LXD

I'm going to be picking on the latest one, mantic. The image name for the mantic version of Ubuntu is mantic-server-cloudimg-amd64-azure.vhd.tar.gz.

Extract the `vhd`

Now that you have that downloaded, extract the vhd file from that archive. You can use WSL to do this, or an application like 7-zip.

7-Zip

Customize the base image

Now that you have the vhd extracted, we need to customize it.

Here is a link to the create-vm.ps1 script that will convert the vhd to a dynamically sizing vhdx, mount the vhdx in WSL, remove the Azure specific components, and finally chroot into the mounted vhdx so you can customize it.

hyperv-cloud-init/set-cloudinit.ps1 at main · EdwardCooke/hyperv-cloud-init

Easy scripts for creating a cloud-init based VM in Hyper-V - hyperv-cloud-init/set-cloudinit.ps1 at main · EdwardCooke/hyperv-cloud-init

GitHubEdwardCooke

You can execute this from where you checked out the repo in an administrator PowerShell window

./set-cloudinit-ubuntu.ps1 -Source C:\cloud-init\livecd.ubuntu-cpc.azure.vhd -Destination C:\cloud-init\ubuntu-mantric.vhdx

You should be greeted with output that looks very similar to this.

Converting image to a dynamic disk
Mounting in WSL
The disk was successfully mounted as '/mnt/wsl/61447da8-c86a-4526-b900-5048ec59d556'.
Note: The location will be different if you have modified the automount.root setting in /etc/wsl.conf.
To unmount and detach the disk, run 'wsl.exe --unmount \\?\C:\cloud-init\ubuntu-mantric.vhdx'.
Modifying the cloud-init data source
Removing the azure specific configuration
Entering your image environment
Press ctrl+d or type exit when you are done customizing your image
root@edslaptop:/#

You are now in the chroot environment of your golden image. Customize as you see fit. Once you are happy with your image, press ctrl+d or type exit and press enter to exit the chroot environment.

After you exit the chroot environment, you should now be greeted with the following

Unmounting from WSL
The operation completed successfully.

Your golden image is now setup and ready to be cloned.

Building your virtual machine

Setup `Cloud-init` files

We're going to now setup some basic example Cloud-init files. You can find the example Cloud-init configuration in the example directory here

hyperv-cloud-init/example at main · EdwardCooke/hyperv-cloud-init

Easy scripts for creating a cloud-init based VM in Hyper-V - hyperv-cloud-init/example at main · EdwardCooke/hyperv-cloud-init

GitHubEdwardCooke

Those example files will show how to build a VM with the hostname of example, a single user with username/password of example and install curl and ca-certificates. It will also enable password based ssh login, which is disabled by default and upgrade all of the currently installed packages.

Use the Cloud-init configurations and build the VM

Now that you have your Cloud-init configuration build, it's time to build the actual virtual machine. We'll use the create-vm.ps1 script for that. It is very simple to use, here is the minimal usage of the script. It creates a virtual machine named Test, with a root disk of 30 gigs, using the example directory, the base image we just created and 4 gigs of memory.

./create-vm.ps1 -VirtualMachineName Test -VirtualDiskSize 30 -CloudInitDirectory ./example -RootDiskPath C:\cloud-init\ubuntu-mantric.vhdx -MemorySize 4

You should now be greeted with an output looking like this.

Copying root disk from C:\cloud-init\ubuntu-mantric.vhdx to C:\ProgramData\Microsoft\Windows\Virtual Hard Disks\Test.vhdx
Resizing virtual disk to 30 gigabytes
Creating cloud init iso at C:\ProgramData\Microsoft\Windows\Virtual Hard Disks\Test-cidata.iso

OSCDIMG 2.56 CD-ROM and DVD-ROM Premastering Utility
Copyright (C) Microsoft, 1993-2012. All rights reserved.
Licensed only for producing Microsoft authorized content.


Scanning source tree
Scanning source tree complete (2 files in 1 directories)

Computing directory information complete

Image file is 57344 bytes

Writing 2 files in 1 directories to C:\ProgramData\Microsoft\Windows\Virtual Hard Disks\Test-cidata.iso

100% complete

Final image file is 57344 bytes

Done.
Creating virtual machine
Disabling dynamic memory
Setting secure boot settings
Adding cidata iso to the virtual machine
Starting virtual machine
Virtual machine creation complete

Check your virtual machine

Open the Hyper-V Manager application and check your Test virtual machine, if it's not the basic login screen then wait a couple of minutes.

Once the login screen is up, try logging in with the example user, the password is also example.

Conclusion

This is the under pinning of how to use Hyper-V, Cloud-init, PowerShell and WSL to customize a base image and actually use it in Hyper-V. Next post will be using PowerShell to easily create a set of Cloud-init files to quickly build a basic Ubuntu virtual machine.

This was a fun project to do. I'm glad I did it, even thought it was a little difficult to figure it all out.

Hopefully you find it useful as well.

Links

cloud-init 23.2.1 documentation

Create and upload an Ubuntu Linux VHD in Azure - Azure Virtual Machines

Learn to create and upload an Azure virtual hard disk (VHD) that contains an Ubuntu Linux operating system.

Microsoft Learnsrijang

Prepare Linux for imaging - Azure Virtual Machines

Learn to prepare a Linux system to be used for an image in Azure.

Microsoft Learnsrijang

GitHub - EdwardCooke/hyperv-cloud-init: Easy scripts for creating a cloud-init based VM in Hyper-V

Easy scripts for creating a cloud-init based VM in Hyper-V - GitHub - EdwardCooke/hyperv-cloud-init: Easy scripts for creating a cloud-init based VM in Hyper-V

GitHubEdwardCooke

Ubuntu Cloud Images - the official Ubuntu images for public clouds, Openstack, KVM and LXD

Ubuntu Cloud Images are the official Ubuntu images that have been customized by Canonical to run on public clouds that provide Ubuntu Certified Images, Openstack, KVM, LXD and more.

the official Ubuntu images for public clouds, Openstack, KVM and LXD

MetalLB, Calico, BGP, and LoadBalancer IPs

Edward Cooke — Wed, 28 Jun 2023 17:37:23 GMT

This post will cover exposing only your LoadBalancer service IP's to your BGP Peer.

Problem statement

Configure MetalLB and Calico to expose your LoadBalancer services to your upstream BGP peered router

Summary

Installing MetalLB is easy, so I'm not going to cover that, I will however, cover the configuration of it and Calico so that it will only expose only the IP's associated with your LoadBalancer services to your BGP peers.

According to the docs, in theory, setting disableBGPExport to true should do what you want, however, that has an unintended side effect. Traffic destined between your pods in your cluster will go out through your default gateway which makes it so pods can't communicate with one another nor the public internet. If you use Linux as your gateway, you may see kernel entries stating IPv4 martian source.

I'm going to show manifests using the 192.168.3.0/24 subnet for your LoadBalancer services, 64512 as your autonomous system ID, and 192.168.0.1 as your peer address. You'll want to update those values to fit your network accordingly. My examples also work for local as your externalTrafficPolicy.

I am not going to cover configuring your upstream router, that is very dependent on what you use and how your network is configured, you'll need to research that if you don't know how to do it.

Another thing you may notice, my manifests use crd.projectcalico.org/v1 and not what the docs from Tigera show, projectcalico.org/v3. The reason being, that API doesn't exist in the installed manifests, I don't know why they show that API in their docs, but, they do.

Solution

Install MetalLB

First step is installing MetalLB, we'll cover the configuration, so go ahead and install it.

You can get that information from the MetalLB site:

MetalLB, bare metal load-balancer for Kubernetes

Configure MetalLB's IP address pool

Next step is creating the IPAddressPool. Here's the manifest for that, it's simple and only contains the addresses option. We don't need, nor want, any BGP configuration for MetalLB. That is all handled by Calico.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - 192.168.3.0/24

Setup a Calico BGP filter

Next up is we need to create a filter for what Calico will send to the upstream BGP peer. Here we will allow everything in the 192.168.3.0/24 subnet.

apiVersion: crd.projectcalico.org/v1
kind: BGPFilter
metadata:
  name: default
spec:
  exportV4:
  - action: Reject
    matchOperator: NotIn
    cidr: 192.168.3.0/24

Setup a Calico BGP peer

Now that we have our filter blocking everything except 192.168.3.0/24 we'll create the BGPPeer, referencing the filter we just created.

apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
  name: default
spec:
  peerIP: 192.168.0.1
  asNumber: 64512
  keepOriginalNextHop: false
  filters:
  - default

Setup the Calico BGP Configuration

Now that we have a peer, create the BGPConfiguration. We add each individual IP in the subnet for a reason. If you do the entire subnet, 192.168.3.0/24 instead, it will push that entire subnet to all your nodes. If you want to use the local type in your service, that won't work since that requires the pod to be on the node that the traffic is coming in on. The externalTrafficPolicy set to local on the service is required to preserve the client IP address. Without that it will show the kubeproxy address running on the node.

For a /24 subnet you can use this handy little bash command to generate those entries, replace the 192.168.3 with your subnet.

for i in $(seq 1 255); do echo "  - cidr: 192.168.3.$i/32"; done

apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  asNumber: 64512
  listenPort: 178
  serviceLoadBalancerIPs:
  - cidr: 192.168.3.0/32
  - cidr: 192.168.3.1/32
  - cidr: 192.168.3.2/32
  - cidr: 192.168.3.3/32
  - cidr: 192.168.3.4/32
  - cidr: 192.168.3.5/32
  - cidr: 192.168.3.6/32
  - cidr: 192.168.3.7/32
  - cidr: 192.168.3.8/32
  - cidr: 192.168.3.9/32
  - cidr: 192.168.3.10/32
  - cidr: 192.168.3.11/32
  - cidr: 192.168.3.12/32
  - cidr: 192.168.3.13/32
  - cidr: 192.168.3.14/32
  - cidr: 192.168.3.15/32
  - cidr: 192.168.3.16/32
  - cidr: 192.168.3.17/32
  - cidr: 192.168.3.18/32
  - cidr: 192.168.3.19/32
  - cidr: 192.168.3.20/32
  - cidr: 192.168.3.21/32
  - cidr: 192.168.3.22/32
  - cidr: 192.168.3.23/32
  - cidr: 192.168.3.24/32
  - cidr: 192.168.3.25/32
  - cidr: 192.168.3.26/32
  - cidr: 192.168.3.27/32
  - cidr: 192.168.3.28/32
  - cidr: 192.168.3.29/32
  - cidr: 192.168.3.30/32
  - cidr: 192.168.3.31/32
  - cidr: 192.168.3.32/32
  - cidr: 192.168.3.33/32
  - cidr: 192.168.3.34/32
  - cidr: 192.168.3.35/32
  - cidr: 192.168.3.36/32
  - cidr: 192.168.3.37/32
  - cidr: 192.168.3.38/32
  - cidr: 192.168.3.39/32
  - cidr: 192.168.3.40/32
  - cidr: 192.168.3.41/32
  - cidr: 192.168.3.42/32
  - cidr: 192.168.3.43/32
  - cidr: 192.168.3.44/32
  - cidr: 192.168.3.45/32
  - cidr: 192.168.3.46/32
  - cidr: 192.168.3.47/32
  - cidr: 192.168.3.48/32
  - cidr: 192.168.3.49/32
  - cidr: 192.168.3.50/32
  - cidr: 192.168.3.51/32
  - cidr: 192.168.3.52/32
  - cidr: 192.168.3.53/32
  - cidr: 192.168.3.54/32
  - cidr: 192.168.3.55/32
  - cidr: 192.168.3.56/32
  - cidr: 192.168.3.57/32
  - cidr: 192.168.3.58/32
  - cidr: 192.168.3.59/32
  - cidr: 192.168.3.60/32
  - cidr: 192.168.3.61/32
  - cidr: 192.168.3.62/32
  - cidr: 192.168.3.63/32
  - cidr: 192.168.3.64/32
  - cidr: 192.168.3.65/32
  - cidr: 192.168.3.66/32
  - cidr: 192.168.3.67/32
  - cidr: 192.168.3.68/32
  - cidr: 192.168.3.69/32
  - cidr: 192.168.3.70/32
  - cidr: 192.168.3.71/32
  - cidr: 192.168.3.72/32
  - cidr: 192.168.3.73/32
  - cidr: 192.168.3.74/32
  - cidr: 192.168.3.75/32
  - cidr: 192.168.3.76/32
  - cidr: 192.168.3.77/32
  - cidr: 192.168.3.78/32
  - cidr: 192.168.3.79/32
  - cidr: 192.168.3.80/32
  - cidr: 192.168.3.81/32
  - cidr: 192.168.3.82/32
  - cidr: 192.168.3.83/32
  - cidr: 192.168.3.84/32
  - cidr: 192.168.3.85/32
  - cidr: 192.168.3.86/32
  - cidr: 192.168.3.87/32
  - cidr: 192.168.3.88/32
  - cidr: 192.168.3.89/32
  - cidr: 192.168.3.90/32
  - cidr: 192.168.3.91/32
  - cidr: 192.168.3.92/32
  - cidr: 192.168.3.93/32
  - cidr: 192.168.3.94/32
  - cidr: 192.168.3.95/32
  - cidr: 192.168.3.96/32
  - cidr: 192.168.3.97/32
  - cidr: 192.168.3.98/32
  - cidr: 192.168.3.99/32
  - cidr: 192.168.3.100/32
  - cidr: 192.168.3.101/32
  - cidr: 192.168.3.102/32
  - cidr: 192.168.3.103/32
  - cidr: 192.168.3.104/32
  - cidr: 192.168.3.105/32
  - cidr: 192.168.3.106/32
  - cidr: 192.168.3.107/32
  - cidr: 192.168.3.108/32
  - cidr: 192.168.3.109/32
  - cidr: 192.168.3.110/32
  - cidr: 192.168.3.111/32
  - cidr: 192.168.3.112/32
  - cidr: 192.168.3.113/32
  - cidr: 192.168.3.114/32
  - cidr: 192.168.3.115/32
  - cidr: 192.168.3.116/32
  - cidr: 192.168.3.117/32
  - cidr: 192.168.3.118/32
  - cidr: 192.168.3.119/32
  - cidr: 192.168.3.120/32
  - cidr: 192.168.3.121/32
  - cidr: 192.168.3.122/32
  - cidr: 192.168.3.123/32
  - cidr: 192.168.3.124/32
  - cidr: 192.168.3.125/32
  - cidr: 192.168.3.126/32
  - cidr: 192.168.3.127/32
  - cidr: 192.168.3.128/32
  - cidr: 192.168.3.129/32
  - cidr: 192.168.3.130/32
  - cidr: 192.168.3.131/32
  - cidr: 192.168.3.132/32
  - cidr: 192.168.3.133/32
  - cidr: 192.168.3.134/32
  - cidr: 192.168.3.135/32
  - cidr: 192.168.3.136/32
  - cidr: 192.168.3.137/32
  - cidr: 192.168.3.138/32
  - cidr: 192.168.3.139/32
  - cidr: 192.168.3.140/32
  - cidr: 192.168.3.141/32
  - cidr: 192.168.3.142/32
  - cidr: 192.168.3.143/32
  - cidr: 192.168.3.144/32
  - cidr: 192.168.3.145/32
  - cidr: 192.168.3.146/32
  - cidr: 192.168.3.147/32
  - cidr: 192.168.3.148/32
  - cidr: 192.168.3.149/32
  - cidr: 192.168.3.150/32
  - cidr: 192.168.3.151/32
  - cidr: 192.168.3.152/32
  - cidr: 192.168.3.153/32
  - cidr: 192.168.3.154/32
  - cidr: 192.168.3.155/32
  - cidr: 192.168.3.156/32
  - cidr: 192.168.3.157/32
  - cidr: 192.168.3.158/32
  - cidr: 192.168.3.159/32
  - cidr: 192.168.3.160/32
  - cidr: 192.168.3.161/32
  - cidr: 192.168.3.162/32
  - cidr: 192.168.3.163/32
  - cidr: 192.168.3.164/32
  - cidr: 192.168.3.165/32
  - cidr: 192.168.3.166/32
  - cidr: 192.168.3.167/32
  - cidr: 192.168.3.168/32
  - cidr: 192.168.3.169/32
  - cidr: 192.168.3.170/32
  - cidr: 192.168.3.171/32
  - cidr: 192.168.3.172/32
  - cidr: 192.168.3.173/32
  - cidr: 192.168.3.174/32
  - cidr: 192.168.3.175/32
  - cidr: 192.168.3.176/32
  - cidr: 192.168.3.177/32
  - cidr: 192.168.3.178/32
  - cidr: 192.168.3.179/32
  - cidr: 192.168.3.180/32
  - cidr: 192.168.3.181/32
  - cidr: 192.168.3.182/32
  - cidr: 192.168.3.183/32
  - cidr: 192.168.3.184/32
  - cidr: 192.168.3.185/32
  - cidr: 192.168.3.186/32
  - cidr: 192.168.3.187/32
  - cidr: 192.168.3.188/32
  - cidr: 192.168.3.189/32
  - cidr: 192.168.3.190/32
  - cidr: 192.168.3.191/32
  - cidr: 192.168.3.192/32
  - cidr: 192.168.3.193/32
  - cidr: 192.168.3.194/32
  - cidr: 192.168.3.195/32
  - cidr: 192.168.3.196/32
  - cidr: 192.168.3.197/32
  - cidr: 192.168.3.198/32
  - cidr: 192.168.3.199/32
  - cidr: 192.168.3.200/32
  - cidr: 192.168.3.201/32
  - cidr: 192.168.3.202/32
  - cidr: 192.168.3.203/32
  - cidr: 192.168.3.204/32
  - cidr: 192.168.3.205/32
  - cidr: 192.168.3.206/32
  - cidr: 192.168.3.207/32
  - cidr: 192.168.3.208/32
  - cidr: 192.168.3.209/32
  - cidr: 192.168.3.210/32
  - cidr: 192.168.3.211/32
  - cidr: 192.168.3.212/32
  - cidr: 192.168.3.213/32
  - cidr: 192.168.3.214/32
  - cidr: 192.168.3.215/32
  - cidr: 192.168.3.216/32
  - cidr: 192.168.3.217/32
  - cidr: 192.168.3.218/32
  - cidr: 192.168.3.219/32
  - cidr: 192.168.3.220/32
  - cidr: 192.168.3.221/32
  - cidr: 192.168.3.222/32
  - cidr: 192.168.3.223/32
  - cidr: 192.168.3.224/32
  - cidr: 192.168.3.225/32
  - cidr: 192.168.3.226/32
  - cidr: 192.168.3.227/32
  - cidr: 192.168.3.228/32
  - cidr: 192.168.3.229/32
  - cidr: 192.168.3.230/32
  - cidr: 192.168.3.231/32
  - cidr: 192.168.3.232/32
  - cidr: 192.168.3.233/32
  - cidr: 192.168.3.234/32
  - cidr: 192.168.3.235/32
  - cidr: 192.168.3.236/32
  - cidr: 192.168.3.237/32
  - cidr: 192.168.3.238/32
  - cidr: 192.168.3.239/32
  - cidr: 192.168.3.240/32
  - cidr: 192.168.3.241/32
  - cidr: 192.168.3.242/32
  - cidr: 192.168.3.243/32
  - cidr: 192.168.3.244/32
  - cidr: 192.168.3.245/32
  - cidr: 192.168.3.246/32
  - cidr: 192.168.3.247/32
  - cidr: 192.168.3.248/32
  - cidr: 192.168.3.249/32
  - cidr: 192.168.3.250/32
  - cidr: 192.168.3.251/32
  - cidr: 192.168.3.252/32
  - cidr: 192.168.3.253/32
  - cidr: 192.168.3.254/32

Configure a service with a local external traffic policy

Here is an example of a local external traffic policy service, this will get assigned an IP address from the allowed pool and that IP will then go up to your BGP peer.

apiVersion: v1
kind: Service
metadata:
  name: lb
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 2368
  selector:
    my: selector
  type: LoadBalancer
  externalTrafficPolicy: Local

Conclusion

That should be all there is to exposing your local LoadBalancer services using MetalLB and Calico with BGP configuring your upstream router.

Setting this up was fun and interesting. I learned a lot about BGP and FRR, that's what I used on my Linux router for this project. Figuring out some of the nuances and inconsistencies in the documentation was a bit challenging, but, that's what these posts are for right?

Links

MetalLB, bare metal load-balancer for Kubernetes

Configure BGP peering | Calico Documentation

Configure BGP peering with full mesh, node-specific peering, ToR, and/or Calico route reflectors.

Calico Documentation

Frakkingsweet

GoBGP with health checks

Purpose

Conclusion

Links

Pre-pull images in the docker-in-docker image

Overview

Solution

Problems

How I did it

Create the buildx build context

Create the Dockerfile

Create the install.sh script

Test

Start it

Validate

Caveat

Conclusion

Links

Cloud-init, Hyper-V and a custom image

Overview

Assumptions with this post

Create the base image

Download

Extract the vhd

Customize the base image

Building your virtual machine

Setup Cloud-init files

Use the Cloud-init configurations and build the VM

Check your virtual machine

Conclusion

Links

MetalLB, Calico, BGP, and LoadBalancer IPs

Problem statement

Summary

Solution

Install MetalLB

Configure MetalLB's IP address pool

Setup a Calico BGP filter

Setup a Calico BGP peer

Setup the Calico BGP Configuration

Configure a service with a local external traffic policy

Conclusion

Links

Create the `buildx` build context

Create the `Dockerfile`

Create the `install.sh` script

Extract the `vhd`

Setup `Cloud-init` files