Running pi-gen on WSL 2
This post is all about building and using a custom kernel for WSL2 and getting pi-gen to work.
I'm tired of running pi-gen on my old slow laptop. Here is how to run it in Windows Subsytem for Linux 2.
This post is all about building and using a custom kernel for WSL2 and getting pi-gen to work.
The first problem I ran in to with running pi-gen for creating a Raspberry Pi image in WSL 2 was that the nbd kernel module did not exist. I was receiving the following error message:
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.4.72-microsoft-standard-WSL2/modules.dep.bin'
modprobe: FATAL: Module nbd not found in directory /lib/modules/5.4.72-microsoft-standard-WSL2
[23:59:27] Unloading image
The second issue is that WSL2 does not like modules. Rightfully so since you would somehow have to get all the modules in to all of your distributions.
The third problem was that pi-gen wants to run modprobe nbd during the build.
The fourth, and final issue, is that binfmt_misc was not being mounted in the runtime container pigen_work.
I am breaking this post in to 2 parts, the custom kernel and pi-gen.
Assumptions
I am making the following assumptions about your environment. If you use something different you will need to adjust accordingly.
- You are using the Debian distribution.
- You are checking out all the git repositories to your home directory.
- You have Docker installed.
- You want to build the
arm64branch of pi-gen to make a 64-bit image for your raspberry pi. - You use VSCode as your file editor with an understanding of how to edit and create files.
- You use PowerShell as your shell on Windows.
Custom Kernel
Let us do the first thing to get this working and get nbd installed in the kernel. To do this, we need to recompile the Linux kernel. This is not as bad as it sounds.
The steps to building a custom kernel are simple.
- Install the build pre-requisites.
- Checkout the official kernel source from Microsoft's git repository.
- Configure the kernel by setting a custom kernel version and enabling the
network block devicemodule. - Build it.
- Configure WSL to use that new kernel.
Install build pre-requisites
There are a few packages we need to install for the build to work. To install the necessary components run this:
sudo apt install libncurses-dev git bc build-essential flex bison libssl-dev libelf-dev python3
Checkout the kernel
We need the source so we can compile it. This will take a while, it is a large repository.
git clone https://github.com/microsoft/WSL2-Linux-Kernel.git
When the repository is cloned, go into the repository directory. We need to be there to check out the correct branch and do the configuration and build.
cd WSL2-Linux-Kernel/
At the time of this writing, the most current kernel is in the linux-msft-wsl-5.4.y branch. You can get a list of the branches available by running git branch -r. That will show all branches in the remote git repository. We will now check out the linux-msft-wsl-5.10.y branch.
git checkout linux-msft-wsl-5.10.y
Configure the kernel
Now that we have the kernel checked out, we need to configure the build configuration to create the network block device driver. Also known as nbd.
The pre-built Microsoft kernel configurations are in the Microsoft directory. We will base our kernel from those configuration files. This way we will keep all the additional optimizations and correct settings that Microsoft uses for the kernel.
make menuconfig KCONFIG_CONFIG=Microsoft/config-wsl
When that starts up you should be presented with a text-based GUI.
We are going to do 2 things in here. First is set a custom kernel version. This will make it easy to tell what kernel it is that we are running so we can validate our work. Second is enable the Network block device support module.
To change the kernel version, press the down arrow until General setup is highlighted and press enter.
Next press the down arrow until the Local version option is highlighted. It is probably the second option down. On mine it was set to -microsoft-standard-WSL2. Press enter.
Change this to whatever you want. Just prefix it with a - so it follows semantic version (semver) standards. I changed mine to -frakkingsweet.
Press tab until Exit is highlighted at the bottom and press enter.
Now that we have a custom kernel version set, we need to enable the Network block device support. Go down to Device Drivers. Press enter. Go to Block devices. Press enter. Go down to Network block device support. Press y so it builds it into the kernel.
Now, in order to build the kernel I also had to disable the Generate BTF typeinfo option. That is under Kernel Hacking-> Compile-time checks and compiler options.
When that option was enabled I was getting the following error:
BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
make: *** [Makefile:1179: vmlinux] Error 1
Press tab until Save is highlighted at the bottom and press enter. Add .new to the end of the file name, so it should end up being Microsoft/config-wsl.new. Press enter to save. Press enter to exit the dialog.
Press tab to highlight Exit and press enter. Do this until you are all the way out.
Build the kernel
Now that the kernel is configured, we need to build it. We will do this by running make with a couple parameters. The first one sets the configuration file to use. We will be using Microsoft/config-wsl.new. The second will specify the number of threads to use. This should be set to the number of cores you have on your computer. I have 12 cores, so I set mine to 12, just change the number to the number of cores you have.
make KCONFIG_CONFIG=Microsoft/config-wsl.new -j12
For me, the build took only a couple of minutes. Be patient though, it is not a small thing that is being built.
If everything works, at the end, you will see something like this:
LD arch/x86/boot/setup.elf
OBJCOPY arch/x86/boot/setup.bin
BUILD arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready (#2)
With the kernel built we need to get it out of your WSL2 environment and copy it to your Windows environment. This way the WSL system can use it. Do that by running the following. Replace <username> with your Windows username.
cp vmlinux /mnt/c/Users/<username>/kernel
Configure WSL2
There are several options you can configure for WSL2, but we are only going to concern ourselves with the kernel. If you want to check out other options, check out the Links section at the bottom of the post.
To configure WSL2, create a new file in your home directory in Windows. Should be something like c:\users\<username> where <username> is your Windows username. I used Visual Studio Code to create the file because it can create files that start with a period. In your Windows PowerShell:
code $ENV:USERPROFILE\.wslconfig
The contents of the file will be the following and, like before, replace <username> with your Windows username:
[wsl2]
kernel=c:\\users\\<username>\\kernel
Note the double backslashes \\. That is required so the system reads them correctly. You can use a forward slash / if you would prefer.
Save and close VSCode.
Now we need to shut down all running WSL2 instances, on your Windows host run the following:
wsl --shutdown --all
Now go back into your WSL system, I am using Debian. So, I just run this in PowerShell.
debian
If all works as expected you should see your kernel name when you run uname -a.
$ uname -a
Linux desktop 5.10.60.1-frakkingsweet+ #4 SMP Fri Nov 5 17:14:36 MDT 2021 x86_64 GNU/Linux
pi-gen
Now we have our custom kernel, and you should be in your WSL2 Linux distribution for running pi-gen. I highly recommend using Docker. I do not know if you can build the image without it on WSL2.
Checkout the source code
Go to wherever you want to check out the pi-gen repository to and clone the repository, I am going to assume it is your home directory.
cd ~
git clone https://github.com/RPi-Distro/pi-gen.git
Now that pi-gen is checked out let's go into it and checkout the arm64 branch.
cd pi-gen
git checkout arm64
Fix modprobe nbd failing
Now that we have the pi-gen code checked out we need to create a config file and fix the modprobe problem. Fire up VSCode in the current directory.
code .
Before we create the config file, modify the scripts/qcow2_handling file and comment out the modprobe nbd max_part=16 line of the file. At the time of writing, it is line 19. This will make it so it can use the nbd driver built that we built in to the kernel.
Fix missing binfmt_misc mount point
Now that we have the modprobe problem fixed, we need to fix the binfmt_misc problem. Edit the build-docker.sh file. We will add 2 lines where it runs the docker image. Right under the following line
bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
Add the following, you will want to add this to right underneath both instances of the above line.
mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc &&
Configure pi-gen
With the build problems fixed we can configure pi-gen. Create a file named config at the root.
Make the contents this:
STAGE_LIST="stage0 stage1 stage2"
That will be a basic bootable image for the Raspberry Pi.
Build pi-gen
Now run build-docker.sh to build the image.
After a while the command should complete, and it should give you some output like how long it took and the image name.
real 13m34.217s
user 0m0.404s
sys 0m0.583s
copying results from deploy/
total 1.8G
drwxr-xr-x 2 edward edward 4.0K Mar 31 20:24 .
drwxr-xr-x 15 edward edward 4.0K Mar 31 20:10 ..
-rw-r--r-- 1 edward edward 1.8G Mar 31 20:24 2021-04-01-ec-lite.img
-rw-r--r-- 1 edward edward 57K Mar 31 20:24 2021-04-01-ec-lite.info
-rw-r--r-- 1 edward edward 5.0K Mar 31 20:24 build.log
pigen_work
Done! Your image(s) should be in deploy/
You can now burn that to your SD card and boot your Pi.
Links
GitHub
GitHub
Conclusion
I wish Microsoft had enabled the nbd module by default. It would have made this so much easier. But, whatever. They did not. It is neat to be able to build my own kernel and use it though. Most likely this is completely unsupported and may break something in the future (I wonder how updates will work...). But so far everything has been working well.
I do not recommend running pi-gen without Docker. There are issues when running on a 64-bit operating system that have been taken care of when using Docker. It also helps keep your system clean from any garbage if a build fails.
It took about an hour and a half to get this working, and it took my build times on my Linux laptop (it is old) from 45 minutes, down to 13 on my desktop. That is a huge improvement and the time taken will quickly be recovered.