Argo, Cert-Manager, Let's Encrypt and Azure DNS

I wanted to use Cert-Manager to manage the certificate for a public site in Kubernetes who's DNS was hosted by Azure.

As I built out this environment in Kubernetes I needed to expose a public site and put an SSL cert in front of it. I chose to use Let's Encrypt for the cert since it's free and Cert-Manager to manage that certificate. I use Azure DNS to host the DNS for the site. Cert-Manager supports doing DNS01 (authorization using DNS TXT records) for Let's Encrypt. I also manage my entire cluster using Argo CD.

Overview

The general overview of all of the steps is:

1. Configure Azure
2. Install Cert-Manager
3. Configure an issuer
4. Configure a certificate
5. Validation

Configure Azure

We'll start configuring Azure by creating an enterprise application. You will need to also create a secret for it as well.

1. Navigate to https://portal.azure.com
2. Navigate to Azure Active Directory
3. In the left pane, go to Enterprise Applications.
4. Click New Application.
5. Click Create your own application.
6. Give it a name, for example, cert-manager.
7. Leave it at the non-gallery option.
8. Click Create.
9. Navigate back to Azure Active Directory.
10. Click App Registrations.
11. Select your application, example, cert-manager.
12. Copy the client id and tenant id from the overview page.
13. Click Certificates & Secrets in the left pane.
14. Click New Client Secret.
15. Give it a name, example cert-manager.
16. Choose an expiration date. You can't do longer than 2 years.
17. Copy the created secret value, you will need it when you configure the cert-manager.

Install Cert-Manager

I used the static manifest deployment option for cert-manager. You could do helm, or whatever. There are a number of ways of installing it. With Argo, and doing static manifests, I could not get Kustomize to work. But since it's all static anyways, it's fine. I just stored the install manifest directly in my repo.

The issue was that Kustomize was breaking the namespaces in the manifest file and cert-manager installs things in both the cert-manager and kube-system namespaces. Sad.

Some of the errors I was getting:

E1008 01:24:20.866641       1 leaderelection.go:325] error retrieving resource lock kube-system/cert-manager-controller: configmaps "cert-manager-controller" is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

E1008 01:28:44.235764       1 leaderelection.go:325] error retrieving resource lock kube-system/cert-manager-cainjector-leader-election: configmaps "cert-manager-cainjector-leader-election" is forbidden: User "system:serviceaccount:cert-manager:cert-manager-cainjector" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

These errors are because the service account and other related role bindings are installed supposed to be installed in the kube-system namespace, not cert-manager which is where Kustomize was trying to put them.

You can look over the different deployment options available out of the box for cert-manager at https://cert-manager.io/docs/installation/.

Configure an issuer

There are some key pieces of information you will need to configure the issuer part of cert-manager. This is what talks to Azure DNS and Let's Encrypt to issue the certificate.

• Service Principal information for the application you just created
• Subscription ID that your DNS zone is in
• Resource group name that your DNS zone is in
• DNS Zone name

First, lets create a secret that contains the secret key for your Azure application. Here is an example YAML file containing such a secret. You can get the base64 encoded version of your secret by piping it through base64. echo "superpassword" | base64

apiVersion: v1
kind: Secret
metadata:
  name: azuredns-config
  namespace: cert-manager
type: Opaque
data:
  client-secret: c3VwZXJwYXNzd29yZAo=

Apply that manifest. If you use Argo, you "could" check it in to your repo, but that's bad, so either use something like Bitnami's Sealed-Secrets or just manually apply that manifest.

Second, lets create the Issuer or ClusterIssuer. The difference is that the Issuer and secret would need to be in the same namespace as the certificate you want to create. ClusterIssuer is not namespace specific and can be referenced without needing to create them in the individual namespace. I chose ClusterIssuer to keep things simpler.

Here is a YAML file containing a cluster issuer, referencing the secret we made above. I'll explain the key parts below

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-example
spec:
  acme:
    server: "https://acme-v02.api.letsencrypt.org/directory"
    email: edward@example.com
    privateKeySecretRef:
      name: letsencrypt-example
    solvers:
      - dns01:
          azureDNS:
            clientID: xxxx-23ed-4d0a-93de-xxxx
            clientSecretSecretRef:
              key: client-secret
              name: azuredns-config
            subscriptionID: xxxx-9de7-47e2-aa70-xxxx
            tenantID: xxxx-09cf-4a19-80c2-xxxx
            resourceGroupName: rg-dns
            hostedZoneName: example.com
            environment: AzurePublicCloud

The parts

Go ahead and apply or check that in to your repository, there's nothing overly sensitive in it.

Configure a Certificate

Now that we have cert-manager installed an issuer configured we can now configure and issue a certificate.

It is a simple manifest containing the requested information for the certificate.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: example-com
spec:
  secretName: example-com
  duration: 2160h0m0s # 90d
  renewBefore: 360h0m0s # 15d
  subject:
    organizations:
      - Example
  commonName: example.com
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
    - example.com
  issuerRef:
    name: letsencrypt-example
    kind: ClusterIssuer

For detailed information about the Certificate object and available options, check out the docs: https://cert-manager.io/docs/usage/certificate/

There were 2 things I had to work through with Argo. The first was the duration and renewBefore. If you notice, my values have 0m0s on the end, 0 minutes and 0 seconds. The docs for cert-manager specify only 2160h. What was happening with Argo is that it was comparing the values with 0m0s with the manifest which originally didn't have. The second was the isCA: false that is in the example in the docs. That's the default and was not showing up when Argo was doing the diff. These 2 problems caused the certificate to always be out of sync in Argo.

Validation

Now check that your certificate is correctly installed, kubectl get secret -n example-com example-com.

You should get something like this:

NAME                      TYPE                                  DATA   AGE
default-token-c2z5x       kubernetes.io/service-account-token   3      93d
example-com               kubernetes.io/tls                     2      90m

Check your certificate object with kubectl get certificate -n example-com. It's READY state should be True

NAME          READY   SECRET        AGE
example-com   True    example-com   92m

Conclusion

This was my first post in a while and it was fun. I enjoyed figuring this out and it's really not very complicated now that I've worked through the various issues. Now hopefully when the certificate gets updated NGINX will pick it up and automatically bring it in for me. I'll find that out in 15 days :)

Links

cert-manager

Automatically provision and manage TLS certificates in Kubernetes

cert-manager

Let’s Encrypt

Let’s Encrypt is a free, automated, and open certificate authority brought to you by the nonprofit Internet Security Research Group (ISRG).

Home

Open source Kubernetes native workflows, events, CI and CD

Argo

DNS | Microsoft Azure

The Microsoft global network of name servers has the scale and redundancy to ensure ultra-high availability and performance for your domains and apps.

Microsoft Azure