I wanted to use Cert-Manager to manage the certificate for a public site in Kubernetes who's DNS was hosted by Azure.
As I built out this environment in Kubernetes I needed to expose a public site and put an SSL cert in front of it. I chose to use Let's Encrypt for the cert since it's free and Cert-Manager to manage that certificate. I use Azure DNS to host the DNS for the site. Cert-Manager supports doing DNS01 (authorization using DNS TXT records) for Let's Encrypt. I also manage my entire cluster using Argo CD.
The general overview of all of the steps is:
We'll start configuring Azure by creating an enterprise application. You will need to also create a secret for it as well.
- Navigate to https://portal.azure.com
- Navigate to
Azure Active Directory
- In the left pane, go to
Create your own application.
- Give it a name, for example,
- Leave it at the
- Navigate back to
Azure Active Directory.
- Select your application, example,
- Copy the
tenant idfrom the overview page.
Certificates & Secretsin the left pane.
New Client Secret.
- Give it a name, example
- Choose an expiration date. You can't do longer than 2 years.
- Copy the created secret value, you will need it when you configure the cert-manager.
I used the static manifest deployment option for cert-manager. You could do helm, or whatever. There are a number of ways of installing it. With Argo, and doing static manifests, I could not get Kustomize to work. But since it's all static anyways, it's fine. I just stored the install manifest directly in my repo.
The issue was that Kustomize was breaking the namespaces in the manifest file and cert-manager installs things in both the
kube-system namespaces. Sad.
Some of the errors I was getting:
E1008 01:24:20.866641 1 leaderelection.go:325] error retrieving resource lock kube-system/cert-manager-controller: configmaps "cert-manager-controller" is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
E1008 01:28:44.235764 1 leaderelection.go:325] error retrieving resource lock kube-system/cert-manager-cainjector-leader-election: configmaps "cert-manager-cainjector-leader-election" is forbidden: User "system:serviceaccount:cert-manager:cert-manager-cainjector" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
These errors are because the service account and other related role bindings are installed supposed to be installed in the
kube-system namespace, not
cert-manager which is where Kustomize was trying to put them.
You can look over the different deployment options available out of the box for cert-manager at https://cert-manager.io/docs/installation/.
Configure an issuer
There are some key pieces of information you will need to configure the issuer part of cert-manager. This is what talks to Azure DNS and Let's Encrypt to issue the certificate.
- Service Principal information for the application you just created
- Subscription ID that your DNS zone is in
- Resource group name that your DNS zone is in
- DNS Zone name
First, lets create a secret that contains the secret key for your Azure application. Here is an example YAML file containing such a secret. You can get the base64 encoded version of your secret by piping it through
echo "superpassword" | base64
apiVersion: v1 kind: Secret metadata: name: azuredns-config namespace: cert-manager type: Opaque data: client-secret: c3VwZXJwYXNzd29yZAo=
Apply that manifest. If you use Argo, you "could" check it in to your repo, but that's bad, so either use something like Bitnami's Sealed-Secrets or just manually apply that manifest.
Second, lets create the
ClusterIssuer. The difference is that the
Issuer and secret would need to be in the same namespace as the certificate you want to create.
ClusterIssuer is not namespace specific and can be referenced without needing to create them in the individual namespace. I chose
ClusterIssuer to keep things simpler.
Here is a YAML file containing a cluster issuer, referencing the secret we made above. I'll explain the key parts below
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-example spec: acme: server: "https://acme-v02.api.letsencrypt.org/directory" email: email@example.com privateKeySecretRef: name: letsencrypt-example solvers: - dns01: azureDNS: clientID: xxxx-23ed-4d0a-93de-xxxx clientSecretSecretRef: key: client-secret name: azuredns-config subscriptionID: xxxx-9de7-47e2-aa70-xxxx tenantID: xxxx-09cf-4a19-80c2-xxxx resourceGroupName: rg-dns hostedZoneName: example.com environment: AzurePublicCloud
||The name of the issuer, you will reference this in your certificate request|
||The API endpoint for Let's Encrypt|
||The email address that Let's Encrypt will send reminder emails to|
||A secret managed by cert-manager to store Let's Encrypt specific stuff|
||The Client ID of the application you created above|
||The key in the secret containing the Azure application secret|
||The name of the secret containing the Azure application secret|
||The Subscription ID containing your Azure DNS Zone|
||Your Azure Tenant ID|
||The resource group containing your Azure DNS Zone|
||Your Azure DNS Zone|
||Defaults to AzurePublicCloud|
Go ahead and apply or check that in to your repository, there's nothing overly sensitive in it.
Configure a Certificate
Now that we have cert-manager installed an issuer configured we can now configure and issue a certificate.
It is a simple manifest containing the requested information for the certificate.
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: example-com namespace: example-com spec: secretName: example-com duration: 2160h0m0s # 90d renewBefore: 360h0m0s # 15d subject: organizations: - Example commonName: example.com privateKey: algorithm: RSA encoding: PKCS1 size: 2048 usages: - server auth - client auth dnsNames: - example.com issuerRef: name: letsencrypt-example kind: ClusterIssuer
For detailed information about the
Certificate object and available options, check out the docs: https://cert-manager.io/docs/usage/certificate/
There were 2 things I had to work through with Argo. The first was the
renewBefore. If you notice, my values have
0m0s on the end, 0 minutes and 0 seconds. The docs for cert-manager specify only
2160h. What was happening with Argo is that it was comparing the values with
0m0s with the manifest which originally didn't have. The second was the
isCA: false that is in the example in the docs. That's the default and was not showing up when Argo was doing the diff. These 2 problems caused the certificate to always be out of sync in Argo.
Now check that your certificate is correctly installed,
kubectl get secret -n example-com example-com.
You should get something like this:
NAME TYPE DATA AGE default-token-c2z5x kubernetes.io/service-account-token 3 93d example-com kubernetes.io/tls 2 90m
Check your certificate object with
kubectl get certificate -n example-com. It's
READY state should be
NAME READY SECRET AGE example-com True example-com 92m
This was my first post in a while and it was fun. I enjoyed figuring this out and it's really not very complicated now that I've worked through the various issues. Now hopefully when the certificate gets updated NGINX will pick it up and automatically bring it in for me. I'll find that out in 15 days :)