How to run the stock Ubuntu Linux kernel on Scaleway using KEXEC and server tags

Scaleway, like many VPS and baremetal server providers, have their own version of the Linux kernel. No matter which Linux distribution you are installing, you get by default this custom version of the Linux kernel. In each case, you can look into /proc/config.gz for the options that were used when compiling your running kernel.

In this post we are seeing how to set up your cloud server (the VPS, not the baremetal server because those are a bit more complicated to set up) on Scaleway to run the stock Ubuntu Linux kernel. In doing so, we get all the benefits of having a stock Ubuntu Linux server (like ZFS support).

Prerequisites

Each Scaleway baremetal server or VPS has a set number of Linux kernels that you can choose from. You cannot add directly your own Linux kernel but you can find an existing kernel that was compiled with the CONFIG_KEXEC option. If you can find such an existing kernel that has this KEXEC support, you boot with that kernel and then you can switch to the Linux kernel of your liking (as shown below).

The cloud servers (VPS) use the paravirtualizated disk driver and the boot device is /dev/vda. However, the baremetal servers use NBD (Network Block Device), which makes it more involved to set up with Ubuntu. The reason is that in the stock Ubuntu Linux kernels, NBD support is compiled as a kernel module (nbd.ko). Therefore, the kernel needs to load the nbd.ko kernel module before attempting to mount the root filesystem. You can add nbd.ko to initramfs, however I then got up to the following error which complains about missing information while trying to mount the root filesystem. I did not go further with that.

The ARM Scaleway servers are based on the Marvel Armada XP. There is KEXEC support in the stock Scaleway Linux kernel, however, my attempts were not fruitful. I did not get any helpful output in the error logs.

In this post, we are demonstrating by using the VS1S VPS server.

and installing Ubuntu 16.04,

Create such a server and come back to continue into the next section.

Checking that CONFIG_KEXEC is present

Run the following command to verify whether the Linux kernel supports CONFIG_KEXEC,

root@scw-b44273:~# zcat /proc/config.gz | grep CONFIG_KEXEC
CONFIG_KEXEC_CORE=y
# CONFIG_KEXEC is not set
CONFIG_KEXEC_FILE=y
# CONFIG_KEXEC_VERIFY_SIG is not set
root@scw-b44273:~#

The issue is, the default Scaleway Linux kernel for the virtual servers does not support CONFIG_KEXEC. It says it supports CONFIG_KEXEC_CORE and CONFIG_KEXEC_FILE, but without CONFIG_KEXEC, it is not enough!

Let’s switch kernel! Go into the administration screen of the server on the Scaleway website, and click on Show to get the Advanced settings,

Once you click on Show, here is what you get,

We are interested in the bootscript. The default Linux kernel is a Linux 4.4.38 version with a configuration that is selected by Scaleway.  Let’s change it,

Here we select 4.10.8 apparmor and click the green tick to save.

Then, we reboot the server by running the following,

root@scw-b44273:~# shutdown -r now
root@scw-b44273:~# Connection to 51.15.79.254 closed by remote host.
Connection to 51.15.79.254 closed.
Exit 255

After a few minutes, the server is rebooted and we connect again with SSH,

Success! This other kernel has CONFIG_KEXEC support.

Installing a stock Ubuntu Linux kernel

First, we select an appropriate stock Ubuntu Linux kernel. Ubuntu 16.04 came initially with Linux 4.4 but later got those Enablement Stacks which upgrade the Linux kernel to newer supported versions. Let’s see what we have in store.

root@scw-b44273:~# apt update
...
root@scw-b44273:~# apt search ^linux-image-virtual
Sorting... Done
Full Text Search... Done
linux-image-virtual/xenial-updates,xenial-security 4.4.0.108.113 amd64
 This package will always depend on the latest minimal generic kernel image.

linux-image-virtual-hwe-16.04/xenial-updates,xenial-security 4.10.0.42.44 amd64
 Virtual Linux kernel image

linux-image-virtual-hwe-16.04-edge/xenial-updates,xenial-security 4.13.0.21.27 amd64
 Virtual Linux kernel image
...

Those linux-image-virtual images are Linux kernel images suitable to run in KVM and such virtualization. That is, they are suitable to run them in a Virtual Private Server (VPS).

There are three packages,

  1. linux-image-virtual, the updated version of the initial Linux 4.4.0 of Ubuntu 16.04.
  2. linux-image-virtual-hwe-16.04, the updated version of Linux 4.10.0, current HWE Enablement Stack for 16.04.
  3. linux-image-virtual-hwe-16.04-edge, the updated version of Linux 4.13, future HWE Enable stack for 16.04.

We select the second (current HWE Enablement Stack), linux-image-virtual-hwe-16.04. We install it, and then update all packages.

NOTE #1: You will be prompted to install GRUB. Select NOT to install GRUB.

NOTE #2: You will be prompted whether to keep the existing configuration of unattended-upgrades. It is safe to keep the existing configuration.

root@scw-b44273:~# apt install linux-image-virtual-hwe-16.04  # if running cloud server
...
root@scw-b44273:~# apt upgrade
...

For completeness, here is the diff for the configuration of unattended-upgrades,

The main differences are

  1. the existing (local version) of the configuration deals only with security upgrades
  2. the existing (local version) blacklists the ndb-client and xndb-client packages, which are the packages that give access to the network storage (network block device, NBD). By blacklisting their unattended upgrade, it makes a policy decision that requires the administrator to upgrade them only explicitly.

Where is the new kernel and initrd?

Here they are, accessible through /vmlinuz and /initrd.img respectively.

Preparing for KEXEC

First, make sure that the package kexec-tools is installed on your system. Those tools are used to invoke the KEXEC syscall in the Linux kernel from the command line. Run the following to install kexec-tools. If you are asked whether kexec-tools should reboot into the kexec-ed Linux kernel, select the default, which is No. The reason is that if kexec does not work, you will have trouble getting back into your server to fix it.

# apt update
...
# apt install kexec-tools
...

While the KEXEC functionality is easy to use, the root filesystem and systemd need some preparation so that the KEXEC Linux kernel manages to boot successfully. If you do not do the following additional preparations in the configuration files, then the server will get stuck when booting.

Clone the repository by https://github.com/BobVul and run the script that updates /etc with 1. entry in /etc/fstab, 2. udev entry for networking and 3. kexec scripts for systemd.

# git clone https://github.com/BobVul/Custom-Kernel-Launcher-for-Scaleway.git
...
# cd Custom-Kernel-Launcher-for-Scaleway/
# ./install.sh

Changes if you run a baremetal server

Note that this tutorial does not have full instructions to get the baremetal server to work with KEXEC. I just add these instructions in case anyone wants to try out. I reached the point of booting the new kernel, however it was not possible to mount the root filesystem with NBD.

If you are trying these instructions on a X86 baremetal Scaleway server, then you need to edit the file /etc/fstab. The instructions in the repository specify a block device that makes sense to the cloud servers (/dev/vda). For the baremetal servers, the block device is /dev/ndb0. Therefore, in that case, edit /etc/fstab to match the device of the root filesystem.

# df /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nbd0 47929956 750472 44721696 2% /
# cat /etc/fstab 
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/nbd0 / ext4 rw,relatime,data=ordered 0 0

Then, perform the following edit for the udev rule to set the correct device for the (physical) network adapter.

# cat /etc/udev/rules.d/50-sckl-network-name.rules 
SUBSYSTEM=="net", DEVPATH=="/devices/pci0000:00/0000:00:14.0/net/*", NAME="eth0"

The Ubuntu Linux kernel does not have NBD support compiled in, but rather it is a kernel module. Therefore, when you boot the Ubuntu Linux kernel, the system is not able to load the nbd.ko kernel module, because the system can only access whatever is found inside the /initrd.img image file. The way to deal with this, is to add nbd.ko in /etc/initrd.img.  You can do that by appending the single word nbd to /etc/initramfs-tools/modules. As in

cat /etc/initramfs-tools/modules

nbd

Finally, run

# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.13.0-37-generic
# lsinitramfs -l /initrd.img | grep nbd
-rw-r--r-- 1 root root 54046 Mar 7 18:26 lib/modules/4.13.0-37-generic/kernel/drivers/block/nbd.ko

The kernel module is indeed there.

Testing that KEXEC works

First, let’s write down the version of the current running kernel. It is the Scaleway kernel that we selected.

# uname -a 
Linux scw-b34c35 4.10.8-apparmor-1 #1 SMP Wed Apr 5 09:42:29 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux #

Then, run the following command. It runs kexec with the appropriate parameters and tries to boot with the new kernel found at /vmlinuz. When you run the command, you will get disconnected from SSH in a few seconds.

# systemctl kexec 
Connection to 93.184.216.34 closed by remote host.

Now, connect again with SSH and check the kernel version.

# uname -a 
Linux scw-b34c35 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux #

This is the new Ubuntu Linux kernel. We have managed to make it get KEXECed over the stock Scaleway Linux kernel, on demand.

If we reboot the server, it would fallback to the stock Scaleway Linux kernel.

Setting the server tags for automated KEXEC

We are ready to set the server tags so that when the server is rebooted, it will boot into the stock Ubuntu kernel. Go back into the server configuration on Scaleway, where it says about TAGS.

Set the TAGS to the following values,

That is, set two tags to KEXEC_KERNEL=/vmlinuz and KEXEC_INITRD=/initrd.img and click on the green tick to Save.

Now restart the server and verify that the Linux kernel gets automatically KEXECed to the new version.

How to revert back to Scaleway Linux kernel

To revert back to the stock Scaleway Linux kernel, simply remove the two server tags and restart your server.

Conclusion

VPS providers only provide certain kernel versions which make it cumbersome. KEXEC is a feature that allows the existing Linux kernel to KEXEC another Linux kernel and get replaced by the new one. In their current version, Scaleway does not have a facility to boot a stock distribution Linux kernel from the Web management settings. Therefore, KEXEC is a sufficient interim feature for now.

By booting the stock Ubuntu Linux kernel, you get ZFS support and proper Apparmor, therefore you can set up easily LXD on Scaleway (no need to compile ZFS anymore).

Permanent link to this article: https://blog.simos.info/how-to-run-the-stock-ubuntu-linux-kernel-on-scaleway-using-kexec-and-server-tags/

7 comments

1 ping

Skip to comment form

  1. Great tutorial thanks ! i had to `apt install kexec-tools` to make it work tho

    1. Thanks fadomire! I added some text on installing kexec-tools in the section Preparing for KEXEC.

  2. Great job and explanations!
    Of course I’m trying now to make it work on a bare metal node. I adapted the scripts from https://github.com/BobVul/Custom-Kernel-Launcher-for-Scaleway
    The net interfaces seem ok when booting through kexec but system fails and falls back to busybock (initramfs) because the main HDD is not found (/dev/nbd0). I guess the LSSD is not mounted as expected but I don’t see how to do that in fstab for instance.
    Any idea?

    1. Thanks!

      I added a new section on how to set up for the baremetal servers. I have not managed to get the Ubuntu kernel to get KEXECed, however I got pretty close to that.

      There is a part that you are missing, therefore read the update to this post. Also, see the error I reached so that you know you have gone as far as I could. Then, you can try to figure out a way to get the root filesystem to mount properly.

  3. Hello from the present!

    I’ve been setting up this on Ubuntu 18.04, I’ve had two issues following this guide and been able to fix them by doing the following:

    1- systemctl kexec does not work anymore
    Instead, use commandkexec -l /vmlinuz –initrd=/initrd.img

    2- No networking present post reboot into new kernel

    I had to turn on DHCP in systemd with:
    systemctl enable systemd-networkd

    Then create a file in
    /etc/systemd/network/20-dhcp.network

    [Match]
    Name=eth0

    [Network]
    DHCP=ipv4

    1. Hi!

      It looks to me that most current images from Scaleway use the stock Ubuntu kernel, hence ZFS should work out of the box. That is certainly try for the VPS offerings, and most likely for the x86 baremetal servers.
      Which Scaleway server did you try and had to resort to KEXEC to make ZFS work?

    • Broonie on September 25, 2020 at 09:30
    • Reply

    @Simos I’m on very old Scaleway VMs that still and can only utilise the bootscripts; my goal was to get Wireguard working

  1. […] but makes it cumbersome because LXD also expects some other features in the Linux kernel. See https://blog.simos.info/how-to-run-the-stock-ubuntu-linux-kernel-on-scaleway-using-kexec-and-server-… on  a trick to run the stock Ubuntu Linux kernel in a Scaleway VPS (not baremetal […]

Leave a Reply to BroonieCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.