You are using LXD from a Linux distribution package and you would like to migrate your existing installation to the Snap LXD package. Let’s do the migration together!
This post is not about live container migration in LXD. Live container migration is about moving a running container from one LXD server to another.
If you do not have LXD installed already, then look for another guide about the installation and set up of LXD from a snap package. A fresh installation of LXD as a snap package is easy.
Note that from the end of 2017, LXD will be generally distributed as a Snap package. If you run LXD 2.0.x from Ubuntu 16.04, you are not affected by this.
Let’s check the version of LXD (Linux distribution package).
$ lxd --version 2.20 $ apt policy lxd lxd: Installed: 2.20-0ubuntu4~16.04.1~ppa1 Candidate: 2.20-0ubuntu4~16.04.1~ppa1 Version table: *** 2.20-0ubuntu4~16.04.1~ppa1 500 500 http://ppa.launchpad.net/ubuntu-lxc/lxd-stable/ubuntu xenial/main amd64 Packages 100 /var/lib/dpkg/status 2.0.11-0ubuntu1~16.04.2 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 2.0.2-0ubuntu1~16.04.1 500 500 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages 2.0.0-0ubuntu4 500 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
In this case, we run LXD version 2.20, and it was installed from the LXD PPA repository.
If you did not enable the LXD PPA repository, you would have an LXD version 2.0.x, the version that was released with Ubuntu 16.04 (what is running above). LXD version 2.0.11 is currently the default version for Ubuntu 16.04.3 and will be supported in that form until 2016 + 5 = 2021. LXD version 2.0.0 is the original LXD version in Ubuntu 16.04 (when original released) and LXD version 2.0.2 is the security update of that LXD 2.0.0.
We are migrating to the LXD snap package. Let’s see how many containers will be migrated.
$ lxc list | grep RUNNING | wc -l 6
It would be a good test to check if something goes horribly wrong.
Let’s check the available incoming LXD snap packages.
$ snap info lxd name: lxd summary: System container manager and API publisher: canonical contact: https://github.com/lxc/lxd/issues description: | LXD is a container manager for system containers. It offers a REST API to remotely manage containers over the network, using an image based workflow and with support for live migration. Images are available for all Ubuntu releases and architectures as well as for a wide number of other Linux distributions. LXD containers are lightweight, secure by default and a great alternative to virtual machines. snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu channels: stable: 2.20 (5182) 44MB - candidate: 2.20 (5182) 44MB - beta: ↑ edge: git-b165982 (5192) 44MB - 2.0/stable: 2.0.11 (4689) 20MB - 2.0/candidate: 2.0.11 (4770) 20MB - 2.0/beta: ↑ 2.0/edge: git-03e9048 (5131) 19MB -
There are several channels to choose from. The stable channel has LXD 2.20, just like the candidate channel. When the LXD 2.21 snap is ready, it will first be released in the candidate channel and stay there for 24 hours. If everything goes well, it will get propagated to the stable channel. LXD 2.20 was released some time ago, that’s why both channels have the same version (at the time of writing this blog post).
There is the edge channel, which has the auto-compiled version from the git source code repository. It is handy to use this channel if you know that a specific fix (that affects you) has been added to the source code, and you want to verify that it actually fixed the issue. Note that the beta channel is not used, therefore it inherits whatever is found in the channel below; the edge channel.
Finally, there are these 2.0/ tagged channels that correspond to the stock 2.0.x LXD versions in Ubuntu 16.04. It looks that those who use the 5-year supported LXD (because Ubuntu 16.04) have the option to switch to a snap version after all.
Installing the LXD snap
Install the LXD snap.
$ snap install lxd lxd 2.20 from 'canonical' installed
Migrating to the LXD snap
Now, the LXD snap is installed, but the DEB/PPA package LXD is the one that is running. We need to run the migration script lxd.migrate that will move the data from the DEB/PPA version over to the Snap version of LXD. In practical terms, it will move files from /var/lib/lxd (old DEB/PPA LXD location), to
$ sudo lxd.migrate => Connecting to source server => Connecting to destination server => Running sanity checks === Source server LXD version: 2.20 LXD PID: 4414 Resources: Containers: 6 Images: 3 Networks: 1 Storage pools: 1 === Destination server LXD version: 2.20 LXD PID: 30329 Resources: Containers: 0 Images: 0 Networks: 0 Storage pools: 0 The migration process will shut down all your containers then move your data to the destination LXD. Once the data is moved, the destination LXD will start and apply any needed updates. And finally your containers will be brought back to their previous state, completing the migration. Are you ready to proceed (yes/no) [default=no]? yes => Shutting down the source LXD => Stopping the source LXD units => Stopping the destination LXD unit => Unmounting source LXD paths => Unmounting destination LXD paths => Wiping destination LXD clean => Moving the data => Moving the database => Backing up the database => Opening the database => Updating the storage backends => Starting the destination LXD => Waiting for LXD to come online === Destination server LXD version: 2.20 LXD PID: 2812 Resources: Containers: 6 Images: 3 Networks: 1 Storage pools: 1 The migration is now complete and your containers should be back online. Do you want to uninstall the old LXD (yes/no) [default=no]? yes All done. You may need to close your current shell and open a new one to have the "lxc" command work.
Testing the migration to the LXD snap
Let’s check that the containers managed to start successfully,
$ lxc list | grep RUNNING | wc -l 6
But let’s check that we can still run Firefox from an LXD container, according to the following post,
Yep, all good. The artifact in the middle (over the c in packaged) is the mouse cursor in wait mode, while GNOME Screenshot is about to take the screenshot. I did not find a report about that in the GNOME Screenshot bugzilla. It is a minor issue and there are several workarounds (1. try one more time, 2. use timer screenshot).
Let’s do some actual testing,
Yep, works as well.
Exploring the LXD snap commands
Let’s type lxd and press Tab.
$ lxd<Tab> lxd lxd.check-kernel lxd.migrate lxd.benchmark lxd.lxc
There are two commands left to try out, lxd.check-kernel and lxd.benchmark. The snap package is called lxd, therefore any additional commands are prepended with lxd.. lxd is the actually LXD server executable. lxd.lxc is the lxc command that we are using for all LXD actions. The LXD snap package makes the appropriate symbolic link so that we just need to write lxc instead of lxd.lxc.
Trying out lxd.check-kernel
Let’s run lxd.check-kernel.
$ sudo lxd.check-kernel Kernel configuration not found at /proc/config.gz; searching... Kernel configuration found at /lib/modules/4.10.0-40-generic/build/.config --- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled newuidmap is not installed newgidmap is not installed Network namespace: enabled --- Control groups --- Cgroups: enabled Cgroup v1 mount points: /sys/fs/cgroup/systemd /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/freezer /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/devices /sys/fs/cgroup/perf_event /sys/fs/cgroup/cpuset /sys/fs/cgroup/hugetlb /sys/fs/cgroup/pids /sys/fs/cgroup/blkio Cgroup v2 mount points: Cgroup v1 clone_children flag: enabled Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: enabled Cgroup cpuset: enabled --- Misc --- Veth pair device: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded Macvlan: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded Vlan: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded Bridges: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded Advanced netfilter: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded CONFIG_NF_NAT_IPV4: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded CONFIG_NF_NAT_IPV6: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded CONFIG_IP_NF_TARGET_MASQUERADE: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded CONFIG_IP6_NF_TARGET_MASQUERADE: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabledmodprobe: ERROR: missing parameters. See -h. , not loadedCONFIG_NETFILTER_XT_MATCH_COMMENT: enabledmodprobe: ERROR: missing parameters. See -h. , not loaded FUSE (for use with lxcfs): enabledmodprobe: ERROR: missing parameters. See -h. , not loaded --- Checkpoint/Restore --- checkpoint restore: enabled CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities: enabled Note : Before booting a new kernel, you can check its configuration usage : CONFIG=/path/to/config /snap/lxd/5182/bin/lxc-checkconfig
This is an important tool if you have issues in getting the LXD to run. In this example in the Misc section, it shows some errors about missing parameters. I suppose they are issues with the tool as the appropriate kernel modules are indeed loaded. My installation of the LXD snap works okay.
Trying out lxd.benchmark
Let’s try out the command without parameters.
$ lxd.benchmark Usage: lxd-benchmark launch [--count=COUNT] [--image=IMAGE] [--privileged=BOOL] [--start=BOOL] [--freeze=BOOL] [--parallel=COUNT] lxd-benchmark start [--parallel=COUNT] lxd-benchmark stop [--parallel=COUNT] lxd-benchmark delete [--parallel=COUNT] --count (= 100) Number of containers to create --freeze (= false) Freeze the container right after start --image (= "ubuntu:") Image to use for the test --parallel (= -1) Number of threads to use --privileged (= false) Use privileged containers --report-file (= "") A CSV file to write test file to. If the file is present, it will be appended to. --report-label (= "") A label for the report entry. By default, the action is used. --start (= true) Start the container after creation error: A valid action (launch, start, stop, delete) must be passed. Exit 1
It is a benchmark tool that allows to create many containers. We can then use the tool to remove those containers. There is an issue with the default number of containers, 100, which is too high. If you run lxd-benchmark launch without specifying a smaller count, you will mess up your LXD installation because you will run out of memory and maybe of disk space. Looking for a bug report… Okay it got buried into this pull request https://github.com/lxc/lxd/pull/3857 and needs to re-open. Ideally, the default count should be 1, and let the user knowingly select a bigger number.
TODO. Here is the new pull request, https://github.com/lxc/lxd/pull/4074
Let’s try carefully lxd-benchmark.
$ lxd.benchmark launch --count 3 Test environment: Server backend: lxd Server version: 2.20 Kernel: Linux Kernel architecture: x86_64 Kernel version: 4.10.0-40-generic Storage backend: zfs Storage version: 0.6.5.9-2 Container backend: lxc Container version: 2.1.1 Test variables: Container count: 3 Container mode: unprivileged Startup mode: normal startup Image: ubuntu: Batches: 0 Batch size: 4 Remainder: 3 [Dec 5 13:24:26.044] Found image in local store: 5f364e2e3f460773a79e9bec2edb5e993d236f035f70267923d43ab22ae3bb62 [Dec 5 13:24:26.044] Batch processing start [Dec 5 13:24:28.817] Batch processing completed in 2.773s
It took just 2.8s to launch then on this computer.
lxd-benchmark launched 3 containers, with names benchmark-%d. Obviously, refrain from using the word benchmark as a name for your own containers. Let’s see these containers
$ lxc list --columns ns4 +---------------+---------+----------------------+ | NAME | STATE | IPV4 | +---------------+---------+----------------------+ | benchmark-1 | RUNNING | 10.52.251.121 (eth0) | +---------------+---------+----------------------+ | benchmark-2 | RUNNING | 10.52.251.20 (eth0) | +---------------+---------+----------------------+ | benchmark-3 | RUNNING | 10.52.251.221 (eth0) | +---------------+---------+----------------------+ ...
Let’s stop them, and finally remove them.
$ lxd.benchmark stop Test environment: Server backend: lxd Server version: 2.20 Kernel: Linux Kernel architecture: x86_64 Kernel version: 4.10.0-40-generic Storage backend: zfs Storage version: 0.6.5.9-2 Container backend: lxc Container version: 2.1.1 [Dec 5 13:31:16.517] Stopping 3 containers [Dec 5 13:31:16.517] Batch processing start [Dec 5 13:31:20.159] Batch processing completed in 3.642s $ lxd.benchmark delete Test environment: Server backend: lxd Server version: 2.20 Kernel: Linux Kernel architecture: x86_64 Kernel version: 4.10.0-40-generic Storage backend: zfs Storage version: 0.6.5.9-2 Container backend: lxc Container version: 2.1.1 [Dec 5 13:31:24.902] Deleting 3 containers [Dec 5 13:31:24.902] Batch processing start [Dec 5 13:31:25.007] Batch processing completed in 0.105s
Note that the lxd-benchmark actions follow the naming of the lxc actions (launch, start, stop and delete).
Error “Target LXD already has images”
$ sudo lxd.migrate => Connecting to source server => Connecting to destination server => Running sanity checks error: Target LXD already has images, aborting. Exit 1
This means that the snap version of LXD has some images and it is not clean. lxd.migrate requires the snap version of LXD to be clean. Solution: remove the LXD snap and install again.
$ snap remove lxd lxd removed $ snap install lxd lxd 2.20 from 'canonical' installed
Which “lxc” command am I running?
This is the lxc command of the DEB/PPA package,
$ which lxc /usr/bin/lxc
This is the lxc command from the LXD snap package.
$ which lxc /snap/bin/lxc
If you installed the LXD snap but you do not see the the /snap/bin/lxc executable, it could be an artifact of your Unix shell. You may have to close that shell window and open a new one.
Error “bash: /usr/bin/lxc: No such file or directory”
If you get the following,
$ which lxc /snap/bin/lxc
but the lxc command is not found,
$ lxc bash: /usr/bin/lxc: No such file or directory Exit 127
then you must close the terminal window and open a new one.
Note: if you loudly refuse to close the current terminal window, you can just type
$ hash -r
which will refresh the list of executables from the $PATH. Applies to bash, zsh. Use rehash if on *csh.
In my case, my filesystem is btrfs, and the /etc/fstab have the user_subvol_rm_allowed added to my partitions to permits delete volumes, this works with lxd from *deb packages but fails with snaps (Ubuntu 16.04.3 my machines…)
Simon, great post, thank you
I have ubuntu 16 lxd on zfs. will the migrate keep my zfs datasets, or will I have to provide a new zfs pool/dataset for the snap lxd, then useless copy over the existing zfs datasets onto new zfs datasets?
What I want is the new snap lxd to use existing ubuntu lxd zfs! with no useless 200GB file copying of containers!
Thanks for the write up. Very helpful.
Worked a treat and now I can use MACVLAN instead of a bridge 🙂
One thing I had to do was run ‘sudo snap install lxd’ on my 16.04.4 server.
Is that right? Should I have been able to run it as my normal user?
To use snap commands without sudo, you would need to log in first with snap into your Launchpad.net account.
Visit https://login.launchpad.net/ and click on I am a new Ubuntu One user to create an account (if you do not have one already).
Then, log in into this account with
$ snap login
Once you do that, then you will be able to run any snap command without sudo.