NFSv4, Kubernetes, nfs-client-provisioner

Getting a Storage Class in Kubernetes with NFSv4 turned out to be relatively simple. I'm using Rancher, Kubernetes and my shared storage is using NFS.

The easiest solution I found to using NFSv4 in Kubernetes was to use the now deprecated nfs-client-provisioner and disable NFSv3 on the server.

For the purpose of this post, I am assuming you have an NFS server setup and running. For your NFS server I am also assuming you are using nfs-kernel-server.

Configure the Kubernetes Storage Class and provisioner

To do this, first install and setup the nfs-client-provisioner. I used the Rancher UI to add it, I had to enable the additional application catalogs for it show up. There is already a bunch of documentation on how to do it with helm or manually if you prefer so I'm not going over those options.

With Rancher, you will need to add 2 custom parameters for it to work, nfs.path and nfs.server. nfs.path gets set to the root of where you want your data stored. nfs.server is the IP or DNS name of your server.

Example:

nfs.path = /volumes/rancher
nfs.server = nfs.example.com

The provisioner works by using a simple mount command, mount -t nfs blah blah to mount the volume. There is no way to specify the mount options to make it use NFSv4. You also need to make sure that the path specified exists. Otherwise the provisioning container will fail to start.

Disable NFSv3

Next up is disable NFSv3 so it clients will use NFSv4. My NFS server is running Debian with nfs-kernel-server. It will probably also work on Ubuntu and other Debian based systems.

Edit the nfs-kernel-server config

sudo vi /etc/default/nfs/nfs-kernel-server

Add --no-nfs-version 3 to the RPCMOUNTDOPTS.

My final nfs-kernel-server file looks like this.

# Number of servers to start up
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information,
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gidsi --no-nfs-version 3

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=""

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=""

Restart NFS using sudo systemctl restart nfs-kernel-server

Conclusion

Storage, and so far, almost everything else in Kubernetes, is a royal pain in the ass when doing it on-premise and not in the cloud. I have yet to understand reasoning behind it, but storage has been the most complicated piece so far. Networking was a piece of cake compared to storage, just use Canal and call that part done.

Since this is a home setup and I cannot afford running anything in the cloud, NFS was the way for me. I also looked into using other solutions that would supposedly work out-of-the-box, like OpenEBS (ha! screw that, its as complicated as Kubernetes) and Longhorn. The insane complexity involved in getting OpenEBS working and maintaining either of them was just too much for a single person who does this in spare time.

There really needs to be a simple way of saying, hey use this mount command, with these options, for a Storage Class. I do not care about dynamic provisioning in my environment, I just want to share a mount with whatever pod needs it. But, that's probably not going to happen because it would just be too simple.

On a side note, it really is a shame that Mirantis bought Docker and, if you read between the lines, said Swarm is going away and to move to Kubernetes. The between the lines is, a Kuberenetes centric company buys Docker and says Swarm will be supported for 2 years....it just does not make business sense to support a system you won't make money from. Kubernetes won the war, I just wish there was a simple way of getting it to work in an on-premise scenario.

As for the storage class provisioner, I found that the option nfs.mountOptions does not work when specified in the StorageClass. It needs to be specified in the claim or volume. Which is dumb because I thought that's what StorageClasses were for. It does not get passed through to the underlying volume or claim when set in the class. The only options that actually get read from the storage class (at least that were actually working for me) are nfs.path and nfs.server. My guess is to read other options the main method in the provisioner needs to be modified to support it. And based on issues and posts I found that seems to be the case. And with it being supposedly deprecated I suspect that will not happen.

Now I need to see what happens when I scale a service to multiple pods, do they all share the same volume, or do they get different volumes, which would make storage even more problematic.

My apologies for the rant, but I needed it.