How to develop the LXD hypervisor – Part #2 (using multipass/KVM VM)

Background: LXD is a hypervisor that manages machine containers on Linux distributions. You install LXD on your Linux distribution and then you can launch machine containers into your distribution running all sort of (other) Linux distributions.

In this post we are having a closer look on how to help in the development of LXD. We are going to develop LXD in a KVM virtual machine (VM) so that when we try both the server and the client,

  1. they will work in a well-defined environment that others can replicate with ease,
  2. it does not affect your desktop computer as the installation is restricted inside the VM

Note that for simple tasks that only involve the LXD client part (command line tool lxc), you can use the simpler guide at

How to develop the LXD hypervisor – Part #1 (client side lxc)

Setting up multipass

See the following guide on installing multipass,

multipass, management of virtual machines running Ubuntu

Then, launch a VM called lxd, that has one CPU core, 2GB RAM, 20GB disk space and runs Ubuntu 16.04. If you plan to use a storage backend with either ZFS or btrfs, then consider that Ubuntu will need 3GB for its own use (including the LXD source).  Therefore, with a 20GB disk, you can allocate 15GB for the storage backend.

$ multipass launch --cpus 1 --disk 20G --mem 2G -n lxd 16.04
Launched: lxd

Let’s see the VM running,

$ multipass list
Name State   IPv4         Release
lxd  RUNNING 10.80.37.120 Ubuntu 16.04 LTS

Get a shell into the VM.

$ multipass exec lxd -- bash --login
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@lxd:~$

We are in. Note the –login parameter. It is essential for snaps to work. There might be some nice simplification here in the future.

Next, install Go. See the following post and install the snap version of Go. Also has instructions on how to set the $GOPATH. Make sure you set properly the $GOPATH in ~/.profile.

Installing the Go programming language in Ubuntu

Now set the $PATH so that includes ~ubuntu/go/bin (location for lxd and lxc compiled executables). This has to be set in both the root and ubuntu users. The reason is that we run the LXD server with sudo. Instead of modifying the ~/.profile for both users, let’s do this systemwide. It’s our VM, so we are entitled to this.

ubuntu@go:~$ sudo nano /etc/profile.d/gobin.sh

Paste the following in the editor

export PATH="$PATH:/home/ubuntu/go/bin"

Then, save and quite. Now, log out and log in again. The new path has been updated.

Now, let’s install Go. As instructed in the post on installing Go, here is the command.

ubuntu@lxd:~$ sudo snap install go --classic
go 1.9.3 from 'mwhudson' installed

Also, remove the pre-installed LXD server and client.

ubuntu@lxd:~$ sudo apt remove --purge lxd lxd-client

Now the VM is ready, and we can get the source of LXD and start compiling.

Compiling LXD

The LXD repository has instructions on how to compile LXD. It has a list of development packages to install. Because we are using multipass with the Ubuntu 16.04 image, it comes already with several of those development packages preinstalled.  Here is the command with the packages that are required.

ubuntu@lxd:~$ sudo apt update
ubuntu@lxd:~$ sudo apt install -y lxc-dev libacl1-dev make pkg-config

Next, there some packages that relate to storage backends. Let’s install them for completeness.

ubuntu@lxd:~$ sudo apt install -y thin-provisioning-tools zfsutils-linux

Getting the source code of LXD

Run the following to get the source code of LXD. We use go get to make a clone of the repository with the source code. -d means to just perform the download, no compiling yet. -v is to be verbose. As we can see, it downloads all dependencies as well.

ubuntu@go:~$ go get -d -v github.com/lxc/lxd/lxd
github.com/lxc/lxd (download)
created GOPATH=/home/ubuntu/go; see 'go help gopath'
github.com/dustinkirkland/golang-petname (download)
github.com/golang/protobuf (download)
github.com/gorilla/mux (download)
...
get "gopkg.in/flosch/pongo2.v3": found meta tag get.metaImport{Prefix:"gopkg.in/flosch/pongo2.v3", VCS:"git", RepoRoot:"https://gopkg.in/flosch/pongo2.v3"} at https://gopkg.in/flosch/pongo2.v3?go-get=1
gopkg.in/flosch/pongo2.v3 (download)
ubuntu@lxd:~$

Now, we can compile. We enter the directory that has the source, and run make.

ubuntu@lxd:~$ cd $GOPATH/src/github.com/lxc/lxd
ubuntu@go:~/go/src/github.com/lxc/lxd$ make
...
LXD built successfully
ubuntu@go:~/go/src/github.com/lxc/lxd$ 

Let’s see now the generated binaries.

ubuntu@go:~/go/src/github.com/lxc/lxd$ ls -l $GOPATH/bin/
total 67872
-rwxrwxr-x 1 ubuntu ubuntu 6546215  Feb 9 13:53 deps
-rwxrwxr-x 1 ubuntu ubuntu 6074448  Feb 9 13:52 fuidshift
-rwxrwxr-x 1 ubuntu ubuntu 12639152 Feb 9 13:52 lxc
-rwxrwxr-x 1 ubuntu ubuntu 25077168 Feb 9 13:53 lxd
-rwxrwxr-x 1 ubuntu ubuntu 9418520  Feb 9 13:53 lxd-benchmark
-rwxrwxr-x 1 ubuntu ubuntu 9731784  Feb 9 13:53 macaroon-identity
ubuntu@go:~/go/src/github.com/lxc/lxd$

Testing the compiled LXD

You need two terminal windows to run LXD. The first will keep running the server, and the second will be to issue commands. Both the lxd init and all other lxc commands.

In the first terminal window, run

ubuntu@lxd:~$ sudo --login lxd --debug --group lxd
INFO[02-09|14:52:43] LXD 2.21 is starting in normal mode path=/var/lib/lxd
INFO[02-09|14:52:43] Kernel uid/gid map: 
INFO[02-09|14:52:43] - u 0 0 4294967295 
INFO[02-09|14:52:43] - g 0 0 4294967295 
INFO[02-09|14:52:43] Configured LXD uid/gid map: 
INFO[02-09|14:52:43] - u 0 100000 65536 
INFO[02-09|14:52:43] - g 0 100000 65536 
WARN[02-09|14:52:43] CGroup memory swap accounting is disabled, swap limits will be ignored. 
DBUG[02-09|14:52:43] No existing storage pools detected. 
INFO[02-09|14:52:43] LXD isn't socket activated 
INFO[02-09|14:52:43] Starting /dev/lxd handler: 
INFO[02-09|14:52:43] - binding devlxd socket socket=/var/lib/lxd/devlxd/sock
INFO[02-09|14:52:43] REST API daemon: 
INFO[02-09|14:52:43] - binding Unix socket socket=/var/lib/lxd/unix.socket
INFO[02-09|14:52:43] Pruning expired images 
INFO[02-09|14:52:43] Done pruning expired images 
INFO[02-09|14:52:43] Expiring log files 
INFO[02-09|14:52:43] Done expiring log files 
INFO[02-09|14:52:43] Updating instance types 
INFO[02-09|14:52:43] Updating images 
INFO[02-09|14:52:43] Done updating images 
INFO[02-09|14:52:47] Done updating instance types

At this point, the server is running and active. The LXD server is running in the foreground and you can view all messages. On the second terminal window, we can issue commands.

Let’s initialize LXD. We accept all the defaults.

ubuntu@lxd:~$ sudo --login lxd init
Do you want to configure a new storage pool (yes/no) [default=yes]? <ENTER>
Name of the new storage pool [default=default]: <ENTER>
Name of the storage backend to use (dir, btrfs, lvm, zfs) [default=zfs]: <ENTER>
Would you like to use an existing block device (yes/no)[default=no]?<ENTER>
Size in GB of the new loop device (15GB minimum): <ENTER>
Would you like LXD to be available over the network (yes/no) [default=no]? <ENTER>
Would you like stale cached images to be updated automatically (yes/no) [default=yes]? <ENTER>
Would you like to create a new network bridge (yes/no) [default=yes]? <ENTER>
What should the new bridge be called [default=lxdbr0]? <ENTER>
What IPv4 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]? <ENTER>
What IPv6 address should be used (CIDR subnet notation, “auto” or “none”) [default=auto]? <ENTER>
LXD has been successfully configured.
ubuntu@lxd:~$

Now we are ready to create containers. Let’s grab a copy of the Alpine Linux (because it is small) so that we can use it later with lxd-benchmark. In the first terminal window, we can always see the verbose messages from the running LXD server.

ubuntu@lxd:~$ lxc image copy images:alpine/3.7 local: --alias alpine --auto-update
Image copied successfully!
ubuntu@lxd:~$

Here is the container image for alpine/3.7, named simply alpine.

ubuntu@lxd:~$ lxc image list
+--------+--------------+--------+-----------------------------------+--------+--------+-----------------------------+
| ALIAS  | FINGERPRINT  | PUBLIC | DESCRIPTION                       | ARCH   | SIZE   | UPLOAD DATE                 |
+--------+--------------+--------+-----------------------------------+--------+--------+-----------------------------+
| alpine | 2c2216b8b844 | no     | Alpine 3.7 amd64 (20180208_17:50) | x86_64 | 1.80MB | Feb 9, 2018 at 1:11pm (UTC) |
+--------+--------------+--------+-----------------------------------+--------+--------+-----------------------------+
ubuntu@lxd:~$

We are good to run the benchmark. It is a binary called lxd-benchmark, available along with lxd and lxc. lxd-benchmark takes launch as subcommand, the image that is available in the local store, and the number of containers to mass-launch.

ubuntu@lxd:~$ lxd-benchmark launch --image=alpine --count=3
Test environment:
 Server backend: lxd
 Server version: 2.21
 Kernel: Linux
 Kernel architecture: x86_64
 Kernel version: 4.4.0-112-generic
 Storage backend: dir
 Storage version: 1
 Container backend: lxc
 Container version: 2.0.8

Test variables:
 Container count: 3
 Container mode: unprivileged
 Startup mode: normal startup
 Image: alpine
 Batches: 3
 Batch size: 1
 Remainder: 0

[Feb 9 15:21:11.300] Found image in local store: 2c2216b8b844bb91f70a1093d511e326b58548b3e03fc1bc1a50c07ee613662f
[Feb 9 15:21:11.300] Batch processing start
[Feb 9 15:21:11.823] Processed 1 containers in 0.523s (1.912/s)
[Feb 9 15:21:12.420] Processed 2 containers in 1.119s (1.787/s)
[Feb 9 15:21:13.041] Batch processing completed in 1.741s
ubuntu@lxd:~$

Let’s see those three containers. They get the automated name benchmark-N. Keep that in mind when you create your own containers so as not to name then like that.

ubuntu@lxd:~$ lxc list --columns ns4tS
+-------------+---------+-----------------------+------------+-----------+
| NAME        | STATE   | IPV4                  | TYPE       | SNAPSHOTS |
+-------------+---------+-----------------------+------------+-----------+
| benchmark-1 | RUNNING | 10.224.164.19 (eth0)  | PERSISTENT | 0         |
+-------------+---------+-----------------------+------------+-----------+
| benchmark-2 | RUNNING | 10.224.164.200 (eth0) | PERSISTENT | 0         |
+-------------+---------+-----------------------+------------+-----------+
| benchmark-3 | RUNNING | 10.224.164.203 (eth0) | PERSISTENT | 0         |
+-------------+---------+-----------------------+------------+-----------+
ubuntu@lxd:~$

Finally, let’s cleanup.

ubuntu@lxd:~$ lxd-benchmark delete
Test environment:
 Server backend: lxd
 Server version: 2.21
 Kernel: Linux
 Kernel architecture: x86_64
 Kernel version: 4.4.0-112-generic
 Storage backend: dir
 Storage version: 1
 Container backend: lxc
 Container version: 2.0.8

[Feb 9 15:25:17.831] Deleting 3 containers
[Feb 9 15:25:17.831] Batch processing start
[Feb 9 15:25:18.548] Processed 1 containers in 0.716s (1.397/s)
[Feb 9 15:25:19.257] Processed 2 containers in 1.425s (1.403/s)
[Feb 9 15:25:19.984] Batch processing completed in 2.152s
ubuntu@lxd:~$

Conclusion

We have set up Ubuntu in a VM with multipass, and made it a development environment for LXD.

We compiled LXD, run it and used lxd-benchmark to demonstrate the creation of several containers.

In a subsequent post, we will see how to make changes to the code.

Troubleshooting

Help! I tried “go get” but I get “can’t load package: package github.com/lxc/lxd: no Go files”

The full error message is

can't load package: package github.com/lxc/lxd: no Go files in /home/ubuntu/go/src/github.com/lxc/lxd

The LXD repository on Github has a few peculiarities. It is a repository that has two Go programs, lxd and lxc. They each reside in their own subdirectories. The main path at github.com/lxc/lxd does not have Go source code; it has a Makefile instead. Therefore, in github.com/lxc/lxd you type make, which then uses Go to compile both lxd/ and lxc/.

Help! I run lxd and it gets stuck with “CGroup memory swap accounting is disabled”

If you do not get any other output apart from that single line, then you forgot to remove the distribution LXD.

ubuntu@lxd:~$ sudo -E $GOPATH/bin/lxd --group lxd
WARN[02-09|13:57:48] CGroup memory swap accounting is disabled, swap limits will be ignored.
Ctrl+C
ubuntu@lxd:~$

Remove with

ubuntu@lxd:~$ sudo apt remove --purge lxd lxd-client

and then restart the VM because there are some lingering commands that are running.

Help! Why only “go get -d -v github.com/lxc/lxd/lxd”? What about the client lxc?

It is a peculiar command, because in github.com/lxc/lxd/lxc there is the code for the client lxc. When are we going to get the client? Well, go get wants a repository that has Go source code. go get requires a repository with Go source files, even if you use it with -d (download only). Therefore, go get clones the full repository, even if you specify just some subdirectory in the repository.

Help! lxd is in the $PATH but it cannot be found with “sudo lxd”! What gives?

We have updated the $PATH in /etc/profile.d/gobin.sh, the ubuntu user can run the compiled executables, but sudo lxd does not run!

The problem is that any files in /etc/profile.d/ are read only if the shell is a login shell. What that means, is that we need to run sudo –login lxd instead. That is, add –login to the command.

Permanent link to this article: https://blog.simos.info/how-to-develop-the-lxd-hypervisor-part-2-using-multipass-kvm-vm/

2 comments

    • John R on February 10, 2018 at 09:38
    • Reply

    Why not just use Proxmox?

    1. I think that Proxmox would be overkill for the development of LXD.

      If you are OK to use the dir storage driver, you can get away with just 4-5GB of disk space for the whole development set up.

Leave a Reply

%d bloggers like this: