MetalLB, Calico, BGP, and LoadBalancer IPs
This post will cover exposing only your LoadBalancer service IP's to your BGP Peer.
Problem statement
Configure MetalLB and Calico to expose your LoadBalancer services to your upstream BGP peered router
Summary
Installing MetalLB is easy, so I'm not going to cover that, I will however, cover the configuration of it and Calico so that it will only expose only the IP's associated with your LoadBalancer services to your BGP peers.
According to the docs, in theory, setting disableBGPExport to true should do what you want, however, that has an unintended side effect. Traffic destined between your pods in your cluster will go out through your default gateway which makes it so pods can't communicate with one another nor the public internet. If you use Linux as your gateway, you may see kernel entries stating IPv4 martian source.
I'm going to show manifests using the 192.168.3.0/24 subnet for your LoadBalancer services, 64512 as your autonomous system ID, and 192.168.0.1 as your peer address. You'll want to update those values to fit your network accordingly. My examples also work for local as your externalTrafficPolicy.
I am not going to cover configuring your upstream router, that is very dependent on what you use and how your network is configured, you'll need to research that if you don't know how to do it.
Another thing you may notice, my manifests use crd.projectcalico.org/v1 and not what the docs from Tigera show, projectcalico.org/v3. The reason being, that API doesn't exist in the installed manifests, I don't know why they show that API in their docs, but, they do.
Solution
Install MetalLB
First step is installing MetalLB, we'll cover the configuration, so go ahead and install it.
You can get that information from the MetalLB site:
Configure MetalLB's IP address pool
Next step is creating the IPAddressPool. Here's the manifest for that, it's simple and only contains the addresses option. We don't need, nor want, any BGP configuration for MetalLB. That is all handled by Calico.
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default
namespace: metallb-system
spec:
addresses:
- 192.168.3.0/24
Setup a Calico BGP filter
Next up is we need to create a filter for what Calico will send to the upstream BGP peer. Here we will allow everything in the 192.168.3.0/24 subnet.
apiVersion: crd.projectcalico.org/v1
kind: BGPFilter
metadata:
name: default
spec:
exportV4:
- action: Reject
matchOperator: NotIn
cidr: 192.168.3.0/24
Setup a Calico BGP peer
Now that we have our filter blocking everything except 192.168.3.0/24 we'll create the BGPPeer, referencing the filter we just created.
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: default
spec:
peerIP: 192.168.0.1
asNumber: 64512
keepOriginalNextHop: false
filters:
- default
Setup the Calico BGP Configuration
Now that we have a peer, create the BGPConfiguration. We add each individual IP in the subnet for a reason. If you do the entire subnet, 192.168.3.0/24 instead, it will push that entire subnet to all your nodes. If you want to use the local type in your service, that won't work since that requires the pod to be on the node that the traffic is coming in on. The externalTrafficPolicy set to local on the service is required to preserve the client IP address. Without that it will show the kubeproxy address running on the node.
For a /24 subnet you can use this handy little bash command to generate those entries, replace the 192.168.3 with your subnet.
for i in $(seq 1 255); do echo " - cidr: 192.168.3.$i/32"; done
apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
asNumber: 64512
listenPort: 178
serviceLoadBalancerIPs:
- cidr: 192.168.3.0/32
- cidr: 192.168.3.1/32
- cidr: 192.168.3.2/32
- cidr: 192.168.3.3/32
- cidr: 192.168.3.4/32
- cidr: 192.168.3.5/32
- cidr: 192.168.3.6/32
- cidr: 192.168.3.7/32
- cidr: 192.168.3.8/32
- cidr: 192.168.3.9/32
- cidr: 192.168.3.10/32
- cidr: 192.168.3.11/32
- cidr: 192.168.3.12/32
- cidr: 192.168.3.13/32
- cidr: 192.168.3.14/32
- cidr: 192.168.3.15/32
- cidr: 192.168.3.16/32
- cidr: 192.168.3.17/32
- cidr: 192.168.3.18/32
- cidr: 192.168.3.19/32
- cidr: 192.168.3.20/32
- cidr: 192.168.3.21/32
- cidr: 192.168.3.22/32
- cidr: 192.168.3.23/32
- cidr: 192.168.3.24/32
- cidr: 192.168.3.25/32
- cidr: 192.168.3.26/32
- cidr: 192.168.3.27/32
- cidr: 192.168.3.28/32
- cidr: 192.168.3.29/32
- cidr: 192.168.3.30/32
- cidr: 192.168.3.31/32
- cidr: 192.168.3.32/32
- cidr: 192.168.3.33/32
- cidr: 192.168.3.34/32
- cidr: 192.168.3.35/32
- cidr: 192.168.3.36/32
- cidr: 192.168.3.37/32
- cidr: 192.168.3.38/32
- cidr: 192.168.3.39/32
- cidr: 192.168.3.40/32
- cidr: 192.168.3.41/32
- cidr: 192.168.3.42/32
- cidr: 192.168.3.43/32
- cidr: 192.168.3.44/32
- cidr: 192.168.3.45/32
- cidr: 192.168.3.46/32
- cidr: 192.168.3.47/32
- cidr: 192.168.3.48/32
- cidr: 192.168.3.49/32
- cidr: 192.168.3.50/32
- cidr: 192.168.3.51/32
- cidr: 192.168.3.52/32
- cidr: 192.168.3.53/32
- cidr: 192.168.3.54/32
- cidr: 192.168.3.55/32
- cidr: 192.168.3.56/32
- cidr: 192.168.3.57/32
- cidr: 192.168.3.58/32
- cidr: 192.168.3.59/32
- cidr: 192.168.3.60/32
- cidr: 192.168.3.61/32
- cidr: 192.168.3.62/32
- cidr: 192.168.3.63/32
- cidr: 192.168.3.64/32
- cidr: 192.168.3.65/32
- cidr: 192.168.3.66/32
- cidr: 192.168.3.67/32
- cidr: 192.168.3.68/32
- cidr: 192.168.3.69/32
- cidr: 192.168.3.70/32
- cidr: 192.168.3.71/32
- cidr: 192.168.3.72/32
- cidr: 192.168.3.73/32
- cidr: 192.168.3.74/32
- cidr: 192.168.3.75/32
- cidr: 192.168.3.76/32
- cidr: 192.168.3.77/32
- cidr: 192.168.3.78/32
- cidr: 192.168.3.79/32
- cidr: 192.168.3.80/32
- cidr: 192.168.3.81/32
- cidr: 192.168.3.82/32
- cidr: 192.168.3.83/32
- cidr: 192.168.3.84/32
- cidr: 192.168.3.85/32
- cidr: 192.168.3.86/32
- cidr: 192.168.3.87/32
- cidr: 192.168.3.88/32
- cidr: 192.168.3.89/32
- cidr: 192.168.3.90/32
- cidr: 192.168.3.91/32
- cidr: 192.168.3.92/32
- cidr: 192.168.3.93/32
- cidr: 192.168.3.94/32
- cidr: 192.168.3.95/32
- cidr: 192.168.3.96/32
- cidr: 192.168.3.97/32
- cidr: 192.168.3.98/32
- cidr: 192.168.3.99/32
- cidr: 192.168.3.100/32
- cidr: 192.168.3.101/32
- cidr: 192.168.3.102/32
- cidr: 192.168.3.103/32
- cidr: 192.168.3.104/32
- cidr: 192.168.3.105/32
- cidr: 192.168.3.106/32
- cidr: 192.168.3.107/32
- cidr: 192.168.3.108/32
- cidr: 192.168.3.109/32
- cidr: 192.168.3.110/32
- cidr: 192.168.3.111/32
- cidr: 192.168.3.112/32
- cidr: 192.168.3.113/32
- cidr: 192.168.3.114/32
- cidr: 192.168.3.115/32
- cidr: 192.168.3.116/32
- cidr: 192.168.3.117/32
- cidr: 192.168.3.118/32
- cidr: 192.168.3.119/32
- cidr: 192.168.3.120/32
- cidr: 192.168.3.121/32
- cidr: 192.168.3.122/32
- cidr: 192.168.3.123/32
- cidr: 192.168.3.124/32
- cidr: 192.168.3.125/32
- cidr: 192.168.3.126/32
- cidr: 192.168.3.127/32
- cidr: 192.168.3.128/32
- cidr: 192.168.3.129/32
- cidr: 192.168.3.130/32
- cidr: 192.168.3.131/32
- cidr: 192.168.3.132/32
- cidr: 192.168.3.133/32
- cidr: 192.168.3.134/32
- cidr: 192.168.3.135/32
- cidr: 192.168.3.136/32
- cidr: 192.168.3.137/32
- cidr: 192.168.3.138/32
- cidr: 192.168.3.139/32
- cidr: 192.168.3.140/32
- cidr: 192.168.3.141/32
- cidr: 192.168.3.142/32
- cidr: 192.168.3.143/32
- cidr: 192.168.3.144/32
- cidr: 192.168.3.145/32
- cidr: 192.168.3.146/32
- cidr: 192.168.3.147/32
- cidr: 192.168.3.148/32
- cidr: 192.168.3.149/32
- cidr: 192.168.3.150/32
- cidr: 192.168.3.151/32
- cidr: 192.168.3.152/32
- cidr: 192.168.3.153/32
- cidr: 192.168.3.154/32
- cidr: 192.168.3.155/32
- cidr: 192.168.3.156/32
- cidr: 192.168.3.157/32
- cidr: 192.168.3.158/32
- cidr: 192.168.3.159/32
- cidr: 192.168.3.160/32
- cidr: 192.168.3.161/32
- cidr: 192.168.3.162/32
- cidr: 192.168.3.163/32
- cidr: 192.168.3.164/32
- cidr: 192.168.3.165/32
- cidr: 192.168.3.166/32
- cidr: 192.168.3.167/32
- cidr: 192.168.3.168/32
- cidr: 192.168.3.169/32
- cidr: 192.168.3.170/32
- cidr: 192.168.3.171/32
- cidr: 192.168.3.172/32
- cidr: 192.168.3.173/32
- cidr: 192.168.3.174/32
- cidr: 192.168.3.175/32
- cidr: 192.168.3.176/32
- cidr: 192.168.3.177/32
- cidr: 192.168.3.178/32
- cidr: 192.168.3.179/32
- cidr: 192.168.3.180/32
- cidr: 192.168.3.181/32
- cidr: 192.168.3.182/32
- cidr: 192.168.3.183/32
- cidr: 192.168.3.184/32
- cidr: 192.168.3.185/32
- cidr: 192.168.3.186/32
- cidr: 192.168.3.187/32
- cidr: 192.168.3.188/32
- cidr: 192.168.3.189/32
- cidr: 192.168.3.190/32
- cidr: 192.168.3.191/32
- cidr: 192.168.3.192/32
- cidr: 192.168.3.193/32
- cidr: 192.168.3.194/32
- cidr: 192.168.3.195/32
- cidr: 192.168.3.196/32
- cidr: 192.168.3.197/32
- cidr: 192.168.3.198/32
- cidr: 192.168.3.199/32
- cidr: 192.168.3.200/32
- cidr: 192.168.3.201/32
- cidr: 192.168.3.202/32
- cidr: 192.168.3.203/32
- cidr: 192.168.3.204/32
- cidr: 192.168.3.205/32
- cidr: 192.168.3.206/32
- cidr: 192.168.3.207/32
- cidr: 192.168.3.208/32
- cidr: 192.168.3.209/32
- cidr: 192.168.3.210/32
- cidr: 192.168.3.211/32
- cidr: 192.168.3.212/32
- cidr: 192.168.3.213/32
- cidr: 192.168.3.214/32
- cidr: 192.168.3.215/32
- cidr: 192.168.3.216/32
- cidr: 192.168.3.217/32
- cidr: 192.168.3.218/32
- cidr: 192.168.3.219/32
- cidr: 192.168.3.220/32
- cidr: 192.168.3.221/32
- cidr: 192.168.3.222/32
- cidr: 192.168.3.223/32
- cidr: 192.168.3.224/32
- cidr: 192.168.3.225/32
- cidr: 192.168.3.226/32
- cidr: 192.168.3.227/32
- cidr: 192.168.3.228/32
- cidr: 192.168.3.229/32
- cidr: 192.168.3.230/32
- cidr: 192.168.3.231/32
- cidr: 192.168.3.232/32
- cidr: 192.168.3.233/32
- cidr: 192.168.3.234/32
- cidr: 192.168.3.235/32
- cidr: 192.168.3.236/32
- cidr: 192.168.3.237/32
- cidr: 192.168.3.238/32
- cidr: 192.168.3.239/32
- cidr: 192.168.3.240/32
- cidr: 192.168.3.241/32
- cidr: 192.168.3.242/32
- cidr: 192.168.3.243/32
- cidr: 192.168.3.244/32
- cidr: 192.168.3.245/32
- cidr: 192.168.3.246/32
- cidr: 192.168.3.247/32
- cidr: 192.168.3.248/32
- cidr: 192.168.3.249/32
- cidr: 192.168.3.250/32
- cidr: 192.168.3.251/32
- cidr: 192.168.3.252/32
- cidr: 192.168.3.253/32
- cidr: 192.168.3.254/32
Configure a service with a local external traffic policy
Here is an example of a local external traffic policy service, this will get assigned an IP address from the allowed pool and that IP will then go up to your BGP peer.
apiVersion: v1
kind: Service
metadata:
name: lb
namespace: default
spec:
ports:
- port: 80
protocol: TCP
targetPort: 2368
selector:
my: selector
type: LoadBalancer
externalTrafficPolicy: Local
Conclusion
That should be all there is to exposing your local LoadBalancer services using MetalLB and Calico with BGP configuring your upstream router.
Setting this up was fun and interesting. I learned a lot about BGP and FRR, that's what I used on my Linux router for this project. Figuring out some of the nuances and inconsistencies in the documentation was a bit challenging, but, that's what these posts are for right?
Links
Calico Documentation