Scaleway, like many VPS and baremetal server providers, have their own version of the Linux kernel. No matter which Linux distribution you are installing, you get by default this custom version of the Linux kernel. In each case, you can look into /proc/config.gz for the options that were used when compiling your running kernel.
In this post we are seeing how to set up your server on Scaleway to run the stock Ubuntu Linux kernel. In doing so, we get all the benefits of having a stock Ubuntu Linux server (like ZFS support).
Each Scaleway baremetal server or VPS has a set number of Linux kernels that you can choose from. You cannot add directly your own Linux kernel but you can find an existing kernel that was compiled with the CONFIG_KEXEC option. If you can find such an existing kernel that has this KEXEC support, you boot with that kernel and then you can switch to the Linux kernel of your liking (as shown below).
The ARM Scaleway servers are based on the Marvel Armada XP. There is KEXEC support in the stock Scaleway Linux kernel, however, my attempts were not fruitful. I did not get any helpful output in the error logs.
In this post, we are demonstrating by using the VS1S VPS server.
and installing Ubuntu 16.04,
Create such a server and come back to continue into the next section.
Checking that CONFIG_KEXEC is present
Run the following command to verify whether the Linux kernel supports CONFIG_KEXEC,
root@scw-b44273:~# zcat /proc/config.gz | grep CONFIG_KEXEC CONFIG_KEXEC_CORE=y # CONFIG_KEXEC is not set CONFIG_KEXEC_FILE=y # CONFIG_KEXEC_VERIFY_SIG is not set root@scw-b44273:~#
The issue is, the default Scaleway Linux kernel does not support CONFIG_KEXEC. It says it supports CONFIG_KEXEC_CORE and CONFIG_KEXEC_FILE, but without CONFIG_KEXEC, it is not enough!
Let’s switch kernel! Go into the administration screen of the server on the Scaleway website, and click on Show to get the Advanced settings,
Once you click on Show, here is what you get,
We are interested in the bootscript. The default Linux kernel is a Linux 4.4.38 version with a configuration that is selected by Scaleway. Let’s change it,
Here we select 4.10.8 apparmor and click the green tick to save.
Then, we reboot the server by running the following,
root@scw-b44273:~# shutdown -r now root@scw-b44273:~# Connection to 188.8.131.52 closed by remote host. Connection to 184.108.40.206 closed. Exit 255
After a few minutes, the server is rebooted and we connect again with SSH,
Success! This other kernel has CONFIG_KEXEC support.
Installing a stock Ubuntu Linux kernel
First, we select an appropriate stock Ubuntu Linux kernel. Ubuntu 16.04 came initially with Linux 4.4 but later got those Enablement Stacks which upgrade the Linux kernel to newer supported versions. Let’s see what we have in store.
root@scw-b44273:~# apt update ... root@scw-b44273:~# apt search ^linux-image-virtual Sorting... Done Full Text Search... Done linux-image-virtual/xenial-updates,xenial-security 220.127.116.11.113 amd64 This package will always depend on the latest minimal generic kernel image. linux-image-virtual-hwe-16.04/xenial-updates,xenial-security 18.104.22.168.44 amd64 Virtual Linux kernel image linux-image-virtual-hwe-16.04-edge/xenial-updates,xenial-security 22.214.171.124.27 amd64 Virtual Linux kernel image ...
Those linux-image-virtual images are Linux kernel images suitable to run in KVM and such virtualization. That is, they are suitable to run them in a Virtual Private Server (VPS).
There are three packages,
- linux-image-virtual, the updated version of the initial Linux 4.4.0 of Ubuntu 16.04.
- linux-image-virtual-hwe-16.04, the updated version of Linux 4.10.0, current HWE Enablement Stack for 16.04.
- linux-image-virtual-hwe-16.04-edge, the updated version of Linux 4.13, future HWE Enable stack for 16.04.
We select the second (current HWE Enablement Stack), linux-image-virtual-hwe-16.04. We install it, and then update all packages.
NOTE #1: You will be prompted to install GRUB. Select NOT to install GRUB.
NOTE #2: You will be prompted whether to keep the existing configuration of unattended-upgrades. It is safe to keep the existing configuration.
root@scw-b44273:~# apt install linux-image-virtual-hwe-16.04 ... root@scw-b44273:~# apt upgrade ...
For completeness, here is the diff for the configuration of unattended-upgrades,
The main differences are
- the existing (local version) of the configuration deals only with security upgrades
- the existing (local version) blacklists the ndb-client and xndb-client packages, which are the packages that give access to the network storage (network block device, NBD). By blacklisting their unattended upgrade, it makes a policy decision that requires the administrator to upgrade them only explicitly.
Where is the new kernel and initrd?
Here they are, accessible through /vmlinuz and /initrd.img respectively.
Preparing for KEXEC
While the KEXEC functionality is easy to use, the root filesystem and systemd need some preparation so that the KEXEC Linux kernel manages to boot successfully. If you do not do the following additional preparations in the configuration files, then the server will get stuck when booting.
Clone the repository by https://github.com/BobVul and run the script that updates /etc with 1. entry in /etc/fstab, 2. udev entry for networking and 3. kexec scripts for systemd.
# git clone https://github.com/BobVul/Custom-Kernel-Launcher-for-Scaleway.git... # cd Custom-Kernel-Launcher-for-Scaleway/ # ./install.sh
Testing that KEXEC works
First, let’s write down the version of the current running kernel. It is the Scaleway kernel that we selected.
# uname -aLinux scw-b34c35 4.10.8-apparmor-1 #1 SMP Wed Apr 5 09:42:29 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux #
Then, run the following command. It runs kexec with the appropriate parameters and tries to boot with the new kernel found at /vmlinuz. When you run the command, you will get disconnected from SSH in a few seconds.
# systemctl kexecConnection to 126.96.36.199 closed by remote host.
Now, connect again with SSH and check the kernel version.
# uname -a Linux scw-b34c35 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux #
This is the new Ubuntu Linux kernel. We have managed to make it get KEXECed over the stock Scaleway Linux kernel, on demand.
If we reboot the server, it would fallback to the stock Scaleway Linux kernel.
Setting the server tags for automated KEXEC
We are ready to set the server tags so that when the server is rebooted, it will boot into the stock Ubuntu kernel. Go back into the server configuration on Scaleway, where it says about TAGS.
Set the TAGS to the following values,
That is, set two tags to KEXEC_KERNEL=/vmlinuz and KEXEC_INITRD=/initrd.img and click on the green tick to Save.
Now restart the server and verify that the Linux kernel gets automatically KEXECed to the new version.
How to revert back to Scaleway Linux kernel
To revert back to the stock Scaleway Linux kernel, simply remove the two server tags and restart your server.
VPS providers only provide certain kernel versions which make it cumbersome. KEXEC is a feature that allows the existing Linux kernel to KEXEC another Linux kernel and get replaced by the new one. In their current version, Scaleway does not have a facility to boot a stock distribution Linux kernel from the Web management settings. Therefore, KEXEC is a sufficient interim feature for now.
By booting the stock Ubuntu Linux kernel, you get ZFS support and proper Apparmor, therefore you can set up easily LXD on Scaleway (no need to compile ZFS anymore).