Scaleway, like many VPS and baremetal server providers, have their own version of the Linux kernel. No matter which Linux distribution you are installing, you get by default this custom version of the Linux kernel. In each case, you can look into /proc/config.gz for the options that were used when compiling your running kernel.
In this post we are seeing how to set up your cloud server (the VPS, not the baremetal server because those are a bit more complicated to set up) on Scaleway to run the stock Ubuntu Linux kernel. In doing so, we get all the benefits of having a stock Ubuntu Linux server (like ZFS support).
Each Scaleway baremetal server or VPS has a set number of Linux kernels that you can choose from. You cannot add directly your own Linux kernel but you can find an existing kernel that was compiled with the CONFIG_KEXEC option. If you can find such an existing kernel that has this KEXEC support, you boot with that kernel and then you can switch to the Linux kernel of your liking (as shown below).
The cloud servers (VPS) use the paravirtualizated disk driver and the boot device is /dev/vda. However, the baremetal servers use NBD (Network Block Device), which makes it more involved to set up with Ubuntu. The reason is that in the stock Ubuntu Linux kernels, NBD support is compiled as a kernel module (nbd.ko). Therefore, the kernel needs to load the nbd.ko kernel module before attempting to mount the root filesystem. You can add nbd.ko to initramfs, however I then got up to the following error which complains about missing information while trying to mount the root filesystem. I did not go further with that.
The ARM Scaleway servers are based on the Marvel Armada XP. There is KEXEC support in the stock Scaleway Linux kernel, however, my attempts were not fruitful. I did not get any helpful output in the error logs.
In this post, we are demonstrating by using the VS1S VPS server.
and installing Ubuntu 16.04,
Create such a server and come back to continue into the next section.
Checking that CONFIG_KEXEC is present
Run the following command to verify whether the Linux kernel supports CONFIG_KEXEC,
root@scw-b44273:~# zcat /proc/config.gz | grep CONFIG_KEXEC CONFIG_KEXEC_CORE=y # CONFIG_KEXEC is not set CONFIG_KEXEC_FILE=y # CONFIG_KEXEC_VERIFY_SIG is not set root@scw-b44273:~#
The issue is, the default Scaleway Linux kernel for the virtual servers does not support CONFIG_KEXEC. It says it supports CONFIG_KEXEC_CORE and CONFIG_KEXEC_FILE, but without CONFIG_KEXEC, it is not enough!
Let’s switch kernel! Go into the administration screen of the server on the Scaleway website, and click on Show to get the Advanced settings,
Once you click on Show, here is what you get,
We are interested in the bootscript. The default Linux kernel is a Linux 4.4.38 version with a configuration that is selected by Scaleway. Let’s change it,
Here we select 4.10.8 apparmor and click the green tick to save.
Then, we reboot the server by running the following,
root@scw-b44273:~# shutdown -r now root@scw-b44273:~# Connection to 188.8.131.52 closed by remote host. Connection to 184.108.40.206 closed. Exit 255
After a few minutes, the server is rebooted and we connect again with SSH,
Success! This other kernel has CONFIG_KEXEC support.
Installing a stock Ubuntu Linux kernel
First, we select an appropriate stock Ubuntu Linux kernel. Ubuntu 16.04 came initially with Linux 4.4 but later got those Enablement Stacks which upgrade the Linux kernel to newer supported versions. Let’s see what we have in store.
root@scw-b44273:~# apt update ... root@scw-b44273:~# apt search ^linux-image-virtual Sorting... Done Full Text Search... Done linux-image-virtual/xenial-updates,xenial-security 220.127.116.11.113 amd64 This package will always depend on the latest minimal generic kernel image. linux-image-virtual-hwe-16.04/xenial-updates,xenial-security 18.104.22.168.44 amd64 Virtual Linux kernel image linux-image-virtual-hwe-16.04-edge/xenial-updates,xenial-security 22.214.171.124.27 amd64 Virtual Linux kernel image ...
Those linux-image-virtual images are Linux kernel images suitable to run in KVM and such virtualization. That is, they are suitable to run them in a Virtual Private Server (VPS).
There are three packages,
- linux-image-virtual, the updated version of the initial Linux 4.4.0 of Ubuntu 16.04.
- linux-image-virtual-hwe-16.04, the updated version of Linux 4.10.0, current HWE Enablement Stack for 16.04.
- linux-image-virtual-hwe-16.04-edge, the updated version of Linux 4.13, future HWE Enable stack for 16.04.
We select the second (current HWE Enablement Stack), linux-image-virtual-hwe-16.04. We install it, and then update all packages.
NOTE #1: You will be prompted to install GRUB. Select NOT to install GRUB.
NOTE #2: You will be prompted whether to keep the existing configuration of unattended-upgrades. It is safe to keep the existing configuration.
root@scw-b44273:~# apt install linux-image-virtual-hwe-16.04 # if running cloud server ... root@scw-b44273:~# apt upgrade ...
For completeness, here is the diff for the configuration of unattended-upgrades,
The main differences are
- the existing (local version) of the configuration deals only with security upgrades
- the existing (local version) blacklists the ndb-client and xndb-client packages, which are the packages that give access to the network storage (network block device, NBD). By blacklisting their unattended upgrade, it makes a policy decision that requires the administrator to upgrade them only explicitly.
Where is the new kernel and initrd?
Here they are, accessible through /vmlinuz and /initrd.img respectively.
Preparing for KEXEC
First, make sure that the package kexec-tools is installed on your system. Those tools are used to invoke the KEXEC syscall in the Linux kernel from the command line. Run the following to install kexec-tools. If you are asked whether kexec-tools should reboot into the kexec-ed Linux kernel, select the default, which is No. The reason is that if kexec does not work, you will have trouble getting back into your server to fix it.
# apt update ... # apt install kexec-tools ...
While the KEXEC functionality is easy to use, the root filesystem and systemd need some preparation so that the KEXEC Linux kernel manages to boot successfully. If you do not do the following additional preparations in the configuration files, then the server will get stuck when booting.
Clone the repository by https://github.com/BobVul and run the script that updates /etc with 1. entry in /etc/fstab, 2. udev entry for networking and 3. kexec scripts for systemd.
# git clone https://github.com/BobVul/Custom-Kernel-Launcher-for-Scaleway.git ... # cd Custom-Kernel-Launcher-for-Scaleway/ # ./install.sh
Changes if you run a baremetal server
Note that this tutorial does not have full instructions to get the baremetal server to work with KEXEC. I just add these instructions in case anyone wants to try out. I reached the point of booting the new kernel, however it was not possible to mount the root filesystem with NBD.
If you are trying these instructions on a X86 baremetal Scaleway server, then you need to edit the file /etc/fstab. The instructions in the repository specify a block device that makes sense to the cloud servers (/dev/vda). For the baremetal servers, the block device is /dev/ndb0. Therefore, in that case, edit /etc/fstab to match the device of the root filesystem.
# df / Filesystem 1K-blocks Used Available Use% Mounted on /dev/nbd0 47929956 750472 44721696 2% / # cat /etc/fstab # UNCONFIGURED FSTAB FOR BASE SYSTEM /dev/nbd0 / ext4 rw,relatime,data=ordered 0 0
Then, perform the following edit for the udev rule to set the correct device for the (physical) network adapter.
# cat /etc/udev/rules.d/50-sckl-network-name.rules SUBSYSTEM=="net", DEVPATH=="/devices/pci0000:00/0000:00:14.0/net/*", NAME="eth0"
The Ubuntu Linux kernel does not have NBD support compiled in, but rather it is a kernel module. Therefore, when you boot the Ubuntu Linux kernel, the system is not able to load the nbd.ko kernel module, because the system can only access whatever is found inside the /initrd.img image file. The way to deal with this, is to add nbd.ko in /etc/initrd.img. You can do that by appending the single word nbd to /etc/initramfs-tools/modules. As in
# update-initramfs -u update-initramfs: Generating /boot/initrd.img-4.13.0-37-generic # lsinitramfs -l /initrd.img | grep nbd -rw-r--r-- 1 root root 54046 Mar 7 18:26 lib/modules/4.13.0-37-generic/kernel/drivers/block/nbd.ko
The kernel module is indeed there.
Testing that KEXEC works
First, let’s write down the version of the current running kernel. It is the Scaleway kernel that we selected.
# uname -a Linux scw-b34c35 4.10.8-apparmor-1 #1 SMP Wed Apr 5 09:42:29 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux #
Then, run the following command. It runs kexec with the appropriate parameters and tries to boot with the new kernel found at /vmlinuz. When you run the command, you will get disconnected from SSH in a few seconds.
# systemctl kexec Connection to 126.96.36.199 closed by remote host.
Now, connect again with SSH and check the kernel version.
# uname -a Linux scw-b34c35 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux #
This is the new Ubuntu Linux kernel. We have managed to make it get KEXECed over the stock Scaleway Linux kernel, on demand.
If we reboot the server, it would fallback to the stock Scaleway Linux kernel.
Setting the server tags for automated KEXEC
We are ready to set the server tags so that when the server is rebooted, it will boot into the stock Ubuntu kernel. Go back into the server configuration on Scaleway, where it says about TAGS.
Set the TAGS to the following values,
That is, set two tags to KEXEC_KERNEL=/vmlinuz and KEXEC_INITRD=/initrd.img and click on the green tick to Save.
Now restart the server and verify that the Linux kernel gets automatically KEXECed to the new version.
How to revert back to Scaleway Linux kernel
To revert back to the stock Scaleway Linux kernel, simply remove the two server tags and restart your server.
VPS providers only provide certain kernel versions which make it cumbersome. KEXEC is a feature that allows the existing Linux kernel to KEXEC another Linux kernel and get replaced by the new one. In their current version, Scaleway does not have a facility to boot a stock distribution Linux kernel from the Web management settings. Therefore, KEXEC is a sufficient interim feature for now.
By booting the stock Ubuntu Linux kernel, you get ZFS support and proper Apparmor, therefore you can set up easily LXD on Scaleway (no need to compile ZFS anymore).
Great tutorial thanks ! i had to `apt install kexec-tools` to make it work tho
Thanks fadomire! I added some text on installing kexec-tools in the section Preparing for KEXEC.
Great job and explanations!
Of course I’m trying now to make it work on a bare metal node. I adapted the scripts from https://github.com/BobVul/Custom-Kernel-Launcher-for-Scaleway
The net interfaces seem ok when booting through kexec but system fails and falls back to busybock (initramfs) because the main HDD is not found (/dev/nbd0). I guess the LSSD is not mounted as expected but I don’t see how to do that in fstab for instance.
I added a new section on how to set up for the baremetal servers. I have not managed to get the Ubuntu kernel to get KEXECed, however I got pretty close to that.
There is a part that you are missing, therefore read the update to this post. Also, see the error I reached so that you know you have gone as far as I could. Then, you can try to figure out a way to get the root filesystem to mount properly.
Hello from the present!
I’ve been setting up this on Ubuntu 18.04, I’ve had two issues following this guide and been able to fix them by doing the following:
1- systemctl kexec does not work anymore
Instead, use commandkexec -l /vmlinuz –initrd=/initrd.img
2- No networking present post reboot into new kernel
I had to turn on DHCP in systemd with:
systemctl enable systemd-networkd
Then create a file in
It looks to me that most current images from Scaleway use the stock Ubuntu kernel, hence ZFS should work out of the box. That is certainly try for the VPS offerings, and most likely for the x86 baremetal servers.
Which Scaleway server did you try and had to resort to KEXEC to make ZFS work?
@Simos I’m on very old Scaleway VMs that still and can only utilise the bootscripts; my goal was to get Wireguard working