I'm playing with GitOps and I'm using Argo CD for my orchestration engine. I want to let it rule itself through the git repository.

One of the things I'm implementing in my new Kubernetes cluster is Argo. I'm going to use it for as much of the cluster as I can. First step is getting it to manage itself. Which was a bigger challenge than expected.

First thing is first, some background. My cluster is a basic Kubernetes cluster setup using kubeadm. It currently has 3 control planes and 2 worker nodes. We'll be using Kustomize to setup Argo and let it rule itself. You will need to have kubectl, kustomize and full access to your Kubernetes cluster. The built-in version of Kustomize in kubectl is not current and will not work. You must use the standalone Kustomize binary.

Now to the meat of this. We are going to create a folder structure with a few files. The structure will look like this:

base
- applications.yaml
- ingress.yaml
- secrets.yaml
overlays
- argocd-cm.yaml
- argocd-secret.yaml
- deploy-command.yaml
- service.yaml
kustomization.yaml

applications.yaml will contain the applications in Argo that will be deployed. My example will only have the Argo app.

ingress.yaml will contain the ingress needed to access the Argo UI since that isn't there by default.

secrets.yaml will contain the needed secrets to connect to my git repository.

argocd-cm.yaml contains the configuration that Argo uses. It contains the list of repositories that Argo can see (among other things).

argocd-secret.yaml contains the admin password.

deploy-command.yaml contains a slight modification so that the UI works through nginx or other ingress controllers.

service.yaml sets the service to be a ClusterIP so it works right with an ingress.

kustomization.yaml contains everything to tie it all together.

applications.yaml

Lets begin with applications.yaml. It contains the individual applications that Argo will manage. There's a number of different options, but we'll stick with the basics for Argo.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
spec:
  project: default
  source:
    repoURL: 'git@ssh.dev.azure.com:v3/example/kubernetes/kubernetes'
    path: argocd
    targetRevision: master
  destination:
    namespace: argocd
    name: in-cluster
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

Simple.

A quick overview, we are setting the repoURL to the url for the git repo. (My example uses Azure DevOps, not GitHub). The path in the repository. In this case argocd. And the targetRevision which is a git ref. I wanted to use the master branch so I set it to master. Default is HEAD for git repositories. We also set the namespace to argocd to make it easier when working with the rest of the Argo documentation.

ingress.yaml

Next up is the ingress.yaml file. It's pretty basic, nothing fancy. Just forwarding the correct domain to the service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  annotations:
    ingress.kubernetes.io/proxy-body-size: 100M
    ingress.kubernetes.io/app-root: "/"
spec:
  rules:
  - host: argo.example.com
    http:
      paths:
      - backend:
          service:
            name: argocd-server
            port:
              number: 443
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - argo.example.com

secrets.yaml

In secrets.yaml we need to put in your ssh key to access the git repository. The sshPrivateKey is the base64 encoded private key of your certificate. You can get that by, in Linux, using the base64 utility. cat mykey | base64

The contents of secrets.yaml is like this (my key is removed):

apiVersion: v1
kind: Secret
metadata:
  name: example-kubernetes-ssh
  namespace: argocd
data:
  sshPrivateKey: |
    line1
    line2
    .
    .
    .
type: Opaque

When putting in the key you can do it all on one line if you want, it will look like sshPrivateKey: biglongbase64encodedprivatekey

When doing multiple lines (which makes it easier to read) be sure to tab over your key 4 spaces so it lines up one indent deeper than the sshPrivateKey.

argocd-cm.yaml

We are going to build the config map now. We'll be setting the list of repositories, so Argo knows what to talk to.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  repositories: |-
    - name: vecc
      sshPrivateKeySecret:
        name: example-kubernetes-ssh
        key: sshPrivateKey
      type: git
      url: git@ssh.dev.azure.com:v3/example/kubernetes/kubernetes

It's pretty basic. Setting the URL and what secret to use to talk to it.

argocd-secret.yaml

The argocd-secret.yaml file contains the admin password. To create the hash, which is a bcrypt hash, we use a simple python one-liner. You will need to make sure you have python 2 and the bcrypt module installed.

python -c 'import bcrypt; print(bcrypt.hashpw("password", bcrypt.gensalt(log_rounds=10)))'

Change password to whatever you want. If you want to generate it some other way, go for it. When it dumps out the result, it starts with $2b$. You will want to change that to $2a$. Argo doesn't seem to like the password when it's a b instead of the a. Example: $2b$10$3JE97YUu... should be $2a$10$3JE97YUu...

Next, take that string and pass it through base64 to get the contents for the secret. You'll get a long string of hex. JDJhJDEwJGxlUjFFY1d....

Here's the contents of argocd-secret.yaml. Replace the value for admin.password with the long value you just got.

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
data:
  admin.password: JDJhJDEwJGxlUjFFY1d....

deploy-command.yaml

The deploy-command.yaml file adds --insecure to the list of arguments that are passed to the UI. This is needed because of the gRPC stuff that Argo uses.

- {op: add,  path: /spec/template/spec/containers/0/command/-, value: --insecure}

service.yaml

This reconfigures the service, so it becomes a cluster ip.

apiVersion: v1
kind: Service
metadata:
  name: argocd-server
spec:
  type: ClusterIP

kustomization.yaml

We glue everything together in kustomization.yaml. We also set the Argo images to use.

A small note on the GitHub URL, it will most likely return a 404 if you try to directly browse to it. The reason this works is that Kustomize uses a library called GoGetter. That library sees the URL is for GitHub then parses it and does a git clone for the correct repository, in this case argoproj/argo-cd. Kustomize then references the directory contents of manifests/cluster-install. The ?ref=v1.8.3 is an argument to GoGetter that tells it to checkout branch v1.8.3.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- base/secrets.yaml
- base/applications.yaml
- base/ingress.yaml
- https://github.com/argoproj/argo-cd/manifests/cluster-install?ref=v1.8.3

patchesStrategicMerge:
- overlays/service.yaml
- overlays/argocd-secret.yaml
- overlays/argocd-cm.yaml

patchesJson6902:
- path: overlays/deploy-command.yaml
  target:
    group: apps
    kind: Deployment
    name: argocd-server
    version: v1

namespace: argocd

images:
- name: argoproj/argocd
  newTag: v1.8.3
- name: quay.io/argoproj/argocd
  newTag: v1.8.3

Apply

With those files created, push them up to your git repository. Argo will need them for its first sync.

Now we need to create the argocd namespace and apply the config to the cluster. Think of it as bootstrapping the cluster.

kubectl create ns argocd
kustomize build | kubectl apply -f -

After you apply the manifests, you should be able to get to your Argo UI and see the cluster sync itself. Modifying the applications.yaml file to include additional apps and pushing up the git repo you will see them show up in Argo after its next sync.

Conclusion

Setting up Argo to manage itself was not the easiest of tasks to figure out. And the lack of documentation in a single place made it even harder. On top of that, the version of kustomize baked into the kubectl command is so out of date that it can not reference an external URL, not reliably anyways. Just use the updated kustomize tool.

I like having Argo manage itself. I really like that things are automated in that I can push up a change and it shows up in my cluster. Now to deal with getting rid of nginx and replace it with Traefik. I'm tired of everything going down while my cluster brings in a change.

Useful links

Compute bcrypt hash from command line
I would like to compute the bcrypt hash of my password. Is there an open source command line tool that would do that ? I would use this hash in the Syncthing configuration file (even if I know from
Declarative Setup - Argo CD - Declarative GitOps CD for Kubernetes
argoproj/argo-cd
Declarative continuous deployment for Kubernetes. Contribute to argoproj/argo-cd development by creating an account on GitHub.
argoproj/argocd-example-apps
Example Apps to Demonstrate Argo CD. Contribute to argoproj/argocd-example-apps development by creating an account on GitHub.
Kustomize - Kubernetes native configuration management