Dec 05 2017

How to migrate LXD from DEB/PPA package to Snap package

You are using LXD from a Linux distribution package and you would like to migrate your existing installation to the Snap LXD package. Let’s do the migration together!

This post is not about live container migration in LXD. Live container migration is about moving a running container from one LXD server to another.

If you do not have LXD installed already, then look for another guide about the installation and set up of LXD from a snap package. A fresh installation of LXD as a snap package is easy.

Note that from the end of 2017, LXD will be generally distributed as a Snap package. If you run LXD 2.0.x from Ubuntu 16.04, you are not affected by this.

Prerequisites

Let’s check the version of LXD (Linux distribution package).

$ lxd --version
2.20

$ apt policy lxd
lxd:
 Installed: 2.20-0ubuntu4~16.04.1~ppa1
 Candidate: 2.20-0ubuntu4~16.04.1~ppa1
 Version table:
*** 2.20-0ubuntu4~16.04.1~ppa1 500
      500 http://ppa.launchpad.net/ubuntu-lxc/lxd-stable/ubuntu xenial/main amd64 Packages
      100 /var/lib/dpkg/status
    2.0.11-0ubuntu1~16.04.2 500
      500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
    2.0.2-0ubuntu1~16.04.1 500
      500 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages
    2.0.0-0ubuntu4 500
      500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

In this case, we run LXD version 2.20, and it was installed from the LXD PPA repository.

If you did not enable the LXD PPA repository, you would have an LXD version 2.0.x, the version that was released with Ubuntu 16.04 (what is running above). LXD version 2.0.11 is currently the default version for Ubuntu 16.04.3 and will be supported in that form until 2016 + 5 = 2021. LXD version 2.0.0 is the original LXD version in Ubuntu 16.04 (when original released) and LXD version 2.0.2 is the security update of that LXD 2.0.0.

We are migrating to the LXD snap package. Let’s see how many containers will be migrated.

$ lxc list | grep RUNNING | wc -l
6

It would be a good test to check if something goes horribly wrong.

Let’s check the available incoming LXD snap packages.

$ snap info lxd
name: lxd
summary: System container manager and API
publisher: canonical
contact: https://github.com/lxc/lxd/issues
description: |
 LXD is a container manager for system containers.
 
 It offers a REST API to remotely manage containers over the network, using an
 image based workflow and with support for live migration.
 
 Images are available for all Ubuntu releases and architectures as well as for
 a wide number of other Linux distributions.
 
 LXD containers are lightweight, secure by default and a great alternative to
 virtual machines.
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
channels: 
 stable: 2.20 (5182) 44MB -
 candidate: 2.20 (5182) 44MB -
 beta: ↑ 
 edge: git-b165982 (5192) 44MB -
 2.0/stable: 2.0.11 (4689) 20MB -
 2.0/candidate: 2.0.11 (4770) 20MB -
 2.0/beta: ↑ 
 2.0/edge: git-03e9048 (5131) 19MB -

There are several channels to choose from. The stable channel has LXD 2.20, just like the candidate channel. When the LXD 2.21 snap is ready, it will first be released in the candidate channel and stay there for 24 hours. If everything goes well, it will get propagated to the stable channel. LXD 2.20 was released some time ago, that’s why both channels have the same version (at the time of writing this blog post).

There is the edge channel, which has the auto-compiled version from the git source code repository. It is handy to use this channel if you know that a specific fix (that affects you) has been added to the source code, and you want to verify that it actually fixed the issue. Note that the beta channel is not used, therefore it inherits whatever is found in the channel below; the edge channel.

Finally, there are these 2.0/ tagged channels that correspond to the stock 2.0.x LXD versions in Ubuntu 16.04. It looks that those who use the 5-year supported LXD (because Ubuntu 16.04) have the option to switch to a snap version after all.

Installing the LXD snap

Install the LXD snap.

$ snap install lxd
lxd 2.20 from 'canonical' installed

Migrating to the LXD snap

Now, the LXD snap is installed, but the DEB/PPA package LXD is the one that is running. We need to run the migration script lxd.migrate that will move the data from the DEB/PPA version over to the Snap version of LXD. In practical terms, it will move files from /var/lib/lxd (old DEB/PPA LXD location), to

$ sudo lxd.migrate 
=> Connecting to source server
=> Connecting to destination server
=> Running sanity checks

=== Source server
LXD version: 2.20
LXD PID: 4414
Resources:
 Containers: 6
 Images: 3
 Networks: 1
 Storage pools: 1

=== Destination server
LXD version: 2.20
LXD PID: 30329
Resources:
 Containers: 0
 Images: 0
 Networks: 0
 Storage pools: 0

The migration process will shut down all your containers then move your data to the destination LXD.
Once the data is moved, the destination LXD will start and apply any needed updates.
And finally your containers will be brought back to their previous state, completing the migration.

Are you ready to proceed (yes/no) [default=no]? yes
=> Shutting down the source LXD
=> Stopping the source LXD units
=> Stopping the destination LXD unit
=> Unmounting source LXD paths
=> Unmounting destination LXD paths
=> Wiping destination LXD clean
=> Moving the data
=> Moving the database
=> Backing up the database
=> Opening the database
=> Updating the storage backends
=> Starting the destination LXD
=> Waiting for LXD to come online

=== Destination server
LXD version: 2.20
LXD PID: 2812
Resources:
 Containers: 6
 Images: 3
 Networks: 1
 Storage pools: 1

The migration is now complete and your containers should be back online.
Do you want to uninstall the old LXD (yes/no) [default=no]? yes

All done. You may need to close your current shell and open a new one to have the "lxc" command work.

Testing the migration to the LXD snap

Let’s check that the containers managed to start successfully,

$ lxc list | grep RUNNING | wc -l
6

But let’s check that we can still run Firefox from an LXD container, according to the following post,

How to run graphics-accelerated GUI apps in LXD containers on your Ubuntu desktop

Yep, all good. The artifact in the middle (over the c in packaged) is the mouse cursor in wait mode, while GNOME Screenshot is about to take the screenshot. I did not find a report about that in the GNOME Screenshot bugzilla. It is a minor issue and there are several workarounds (1. try one more time, 2. use timer screenshot).

Let’s do some actual testing,

Yep, works as well.

Exploring the LXD snap commands

Let’s type lxd and press Tab.

$ lxd<Tab>
lxd lxd.check-kernel lxd.migrate 
lxd.benchmark lxd.lxc

There are two commands left to try out, lxd.check-kernel and lxd.benchmark. The snap package is called lxd, therefore any additional commands are prepended with lxd.. lxd is the actually LXD server executable. lxd.lxc is the lxc command that we are using for all LXD actions. The LXD snap package makes the appropriate symbolic link so that we just need to write lxc instead of lxd.lxc.

Trying out lxd.check-kernel

Let’s run lxd.check-kernel.

$ sudo lxd.check-kernel
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /lib/modules/4.10.0-40-generic/build/.config
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 
/sys/fs/cgroup/systemd
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/freezer
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/memory
/sys/fs/cgroup/devices
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/pids
/sys/fs/cgroup/blkio

Cgroup v2 mount points:


Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
Macvlan: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
Vlan: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
Bridges: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
Advanced netfilter: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
CONFIG_NF_NAT_IPV4: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
CONFIG_NF_NAT_IPV6: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabledmodprobe: ERROR: missing parameters. See -h.
, not loadedCONFIG_NETFILTER_XT_MATCH_COMMENT: enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded
FUSE (for use with lxcfs): enabledmodprobe: ERROR: missing parameters. See -h.
, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /snap/lxd/5182/bin/lxc-checkconfig

This is an important tool if you have issues in getting the LXD to run. In this example in the Misc section, it shows some errors about missing parameters. I suppose they are issues with the tool as the appropriate kernel modules are indeed loaded. My installation of the LXD snap works okay.

Trying out lxd.benchmark

Let’s try out the command without parameters.

$ lxd.benchmark 
Usage: lxd-benchmark launch [--count=COUNT] [--image=IMAGE] [--privileged=BOOL] [--start=BOOL] [--freeze=BOOL] [--parallel=COUNT]
 lxd-benchmark start [--parallel=COUNT]
 lxd-benchmark stop [--parallel=COUNT]
 lxd-benchmark delete [--parallel=COUNT]

--count (= 100)
 Number of containers to create
 --freeze (= false)
 Freeze the container right after start
 --image (= "ubuntu:")
 Image to use for the test
 --parallel (= -1)
 Number of threads to use
 --privileged (= false)
 Use privileged containers
 --report-file (= "")
 A CSV file to write test file to. If the file is present, it will be appended to.
 --report-label (= "")
 A label for the report entry. By default, the action is used.
 --start (= true)
 Start the container after creation

error: A valid action (launch, start, stop, delete) must be passed.
Exit 1

It is a benchmark tool that allows to create many containers. We can then use the tool to remove those containers. There is an issue with the default number of containers, 100, which is too high. If you run lxd-benchmark launch without specifying a smaller count,  you will mess up your LXD installation because you will run out of memory and maybe of disk space. Looking for a bug report… Okay it got buried into this pull request https://github.com/lxc/lxd/pull/3857 and needs to re-open. Ideally, the default count should be 1, and let the user knowingly select a bigger number. TODO. Here is the new pull request, https://github.com/lxc/lxd/pull/4074

Let’s try carefully lxd-benchmark.

$ lxd.benchmark launch --count 3
Test environment:
 Server backend: lxd
 Server version: 2.20
 Kernel: Linux
 Kernel architecture: x86_64
 Kernel version: 4.10.0-40-generic
 Storage backend: zfs
 Storage version: 0.6.5.9-2
 Container backend: lxc
 Container version: 2.1.1

Test variables:
 Container count: 3
 Container mode: unprivileged
 Startup mode: normal startup
 Image: ubuntu:
 Batches: 0
 Batch size: 4
 Remainder: 3

[Dec 5 13:24:26.044] Found image in local store: 5f364e2e3f460773a79e9bec2edb5e993d236f035f70267923d43ab22ae3bb62
[Dec 5 13:24:26.044] Batch processing start
[Dec 5 13:24:28.817] Batch processing completed in 2.773s

It took just 2.8s to launch then on this computer.
lxd-benchmark
launched 3 containers, with names benchmark-%d. Obviously, refrain from using the word benchmark as a name for your own containers. Let’s see these containers

$ lxc list --columns ns4
+---------------+---------+----------------------+
| NAME          | STATE   | IPV4                 |
+---------------+---------+----------------------+
| benchmark-1   | RUNNING | 10.52.251.121 (eth0) |
+---------------+---------+----------------------+
| benchmark-2   | RUNNING | 10.52.251.20 (eth0)  |
+---------------+---------+----------------------+
| benchmark-3   | RUNNING | 10.52.251.221 (eth0) |
+---------------+---------+----------------------+
...

Let’s stop them, and finally remove them.

$ lxd.benchmark stop
Test environment:
 Server backend: lxd
 Server version: 2.20
 Kernel: Linux
 Kernel architecture: x86_64
 Kernel version: 4.10.0-40-generic
 Storage backend: zfs
 Storage version: 0.6.5.9-2
 Container backend: lxc
 Container version: 2.1.1

[Dec 5 13:31:16.517] Stopping 3 containers
[Dec 5 13:31:16.517] Batch processing start
[Dec 5 13:31:20.159] Batch processing completed in 3.642s

$ lxd.benchmark delete
Test environment:
 Server backend: lxd
 Server version: 2.20
 Kernel: Linux
 Kernel architecture: x86_64
 Kernel version: 4.10.0-40-generic
 Storage backend: zfs
 Storage version: 0.6.5.9-2
 Container backend: lxc
 Container version: 2.1.1

[Dec 5 13:31:24.902] Deleting 3 containers
[Dec 5 13:31:24.902] Batch processing start
[Dec 5 13:31:25.007] Batch processing completed in 0.105s

Note that the lxd-benchmark actions follow the naming of the lxc actions (launch, start, stop and delete).

Troubleshooting

Error “Target LXD already has images”

$ sudo lxd.migrate 
=> Connecting to source server
=> Connecting to destination server
=> Running sanity checks
error: Target LXD already has images, aborting.
Exit 1

This means that the snap version of LXD has some images and it is not clean. lxd.migrate requires the snap version of LXD to be clean. Solution: remove the LXD snap and install again.

$ snap remove lxd
lxd removed

$ snap install lxd
lxd 2.20 from 'canonical' installed

Which “lxc” command am I running?

This is the lxc command of the DEB/PPA package,

$ which lxc
/usr/bin/lxc

This is the lxc command from the LXD snap package.

$ which lxc
/snap/bin/lxc

If you installed the LXD snap but you do not see the the /snap/bin/lxc executable, it could be an artifact of your Unix shell. You may have to close that shell window and open a new one.

Error “bash: /usr/bin/lxc: No such file or directory”

If you get the following,

$ which lxc
/snap/bin/lxc

but the lxc command is not found,

$ lxc
bash: /usr/bin/lxc: No such file or directory
Exit 127

then you must close the terminal window and open a new one.

Note: if you loudly refuse to close the current terminal window, you can just type

$ hash -r

which will refresh the list of executables from the $PATH. Applies to bash, zsh. Use rehash if on *csh.

 

Permanent link to this article: https://blog.simos.info/how-to-migrate-lxd-from-deb-ppa-package-to-snap-package/

2 comments

1 ping

  1. In my case, my filesystem is btrfs, and the /etc/fstab have the user_subvol_rm_allowed added to my partitions to permits delete volumes, this works with lxd from *deb packages but fails with snaps (Ubuntu 16.04.3 my machines…)

    • MG on December 15, 2017 at 09:28
    • Reply

    Simon, great post, thank you

  1. […] How to migrate LXD from DEB/PPA package to Snap package […]

Leave a Reply

%d bloggers like this: