How to use virtual machines in LXD

Update 22 May 2020: The Ubuntu container images have been updated to install on first boot the LXD Agent in the VM. The corresponding section below has been updated so that you can skip the manual step, if your VM image does it for you.

Update 22 May 2020: See also the tutorial at https://discuss.linuxcontainers.org/t/running-virtual-machines-with-lxd-4-0/7519 by the maintainer of LXD. Ideally, follow both tutorials to get a bigger view of the picture.

Traditionally, LXD is used to create system containers, light-weight virtual machines that use Linux Container features and not hardware virtualization.

However, starting from LXD 3.19, it is possible to create virtual machines as well. That is, now with LXD you can create both system containers and virtual machines.

In the following we see how to setup LXD for virtual machines, then start a virtual machine and use it. Finally, we go through some troubleshooting.

How to setup LXD for virtual machines

Launching LXD virtual machines requires some preparation. We need to pass some information to the virtual machine so that we can then be able to connect to it as soon as it boots up. We pass the necessary information to the virtual machine using a LXD profile, through cloud-init.

Creating a LXD profile for virtual machines

Here is such a profile. There is a cloud-init configuration that essentially has all the information that is passed to the virtual machine. Then, there is a config device that makes available a disk device to the virtual machine, and from there it can setup a VM-specific LXD component.

config:
   user.user-data: |
     #cloud-config
     ssh_pwauth: yes
     
     users:
       - name: ubuntu
         passwd: "$6$iBF0eT1/6UPE2u$V66Rk2BMkR09pHTzW2F.4GHYp3Mb8eu81Sy9srZf5sVzHRNpHP99JhdXEVeN0nvjxXVmoA6lcVEhOOqWEd3Wm0"
         lock_passwd: false
         groups: lxd
         shell: /bin/bash
         sudo: ALL=(ALL) NOPASSWD:ALL
description: LXD profile for virtual machines
devices:
  config:
    source: cloud-init:config
    type: disk
name: vm
used_by:

This profile

  • Enables password authentication in SSH (ssh_pwauth: yes)
  • Adds a non-root user ubuntu with password ubuntu. See Troubleshooting below on how to change this.
  • The password is not in a locked state.
  • The user account belongs to the lxd group, in case we want to run LXD inside the LXD virtual machine.
  • The shell is /bin/bash.
  • Can sudo to all without requiring a password.
  • Some extra configuration will be passed to the virtual machine through an ISO image named config.iso. Once you get a shell in the virtual machine, you can install the rest of the support by mounting this ISO image and running the installer.

We now need to create a profile with the above content. Here is how we do this. You first create an empty profile called vm. Then, you run the cat | lxc profile edit vm command which allows you to paste the above profile configuration and finally hit Control+D to have it saved. Alternatively, you can run lxc profile edit vm and then paste in there the following text. The profile was adapted from the LXD 3.19 announcement page.

$ lxc profile create vm
$ cat | lxc profile edit vm
config:
   user.user-data: |
     #cloud-config
     ssh_pwauth: yes
     
     users:
       - name: ubuntu
         passwd: "$6$iBF0eT1/6UPE2u$V66Rk2BMkR09pHTzW2F.4GHYp3Mb8eu81Sy9srZf5sVzHRNpHP99JhdXEVeN0nvjxXVmoA6lcVEhOOqWEd3Wm0"
         lock_passwd: false
         groups: lxd
         shell: /bin/bash
         sudo: ALL=(ALL) NOPASSWD:ALL
description: LXD profile for virtual machines
devices:
  config:
    source: cloud-init:config
    type: disk
name: vm
used_by:

Ctrl^D
$ lxc profile show vm

We have created the profile with the virtual machine-specific. We have now the pieces in place to launch a LXD virtual machine.

Launching a LXD virtual machine

We launch a LXD virtual machine with the following command. It is the standard lxc launch command, with the addition of the --vm option to create a virtual machine (instead of a system container). We specify the default profile (whichever base configuration you use in your LXD installation) and on top of that we add our VM-specific configuration with --profile vm. Depending on your computer’s specifications, it takes a few seconds to launch the container, and then less than 10 seconds for the VM to boot up and receive the IP address from your network.

$ lxc launch ubuntu:18.04 vm1 --vm --profile default --profile vm
Creating vm1
Starting vm1
$ lxc list vm1
+------+---------+------+------+-----------------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+------+---------+------+------+-----------------+-----------+
| vm1  | RUNNING |      |      | VIRTUAL-MACHINE | 0         |
+------+---------+------+------+-----------------+-----------+
$ lxc list vm1
+------+---------+--------------------+------+-----------------+-----------+
| NAME |  STATE  |        IPV4        | IPV6 |      TYPE       | SNAPSHOTS |
+------+---------+--------------------+------+-----------------+-----------+
| vm1  | RUNNING | 10.10.10.20 (eth0) |      | VIRTUAL-MACHINE | 0         |
+------+---------+--------------------+------+-----------------+-----------+
$ 

We have enabled password authentication for SSH, which means that we can connect to the VM straight away with the following command.

$ ssh ubuntu@10.10.10.20
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-74-generic x86_64)

* Documentation:  https://help.ubuntu.com
* Management:     https://landscape.canonical.com
* Support:        https://ubuntu.com/advantage 

System information as of Fri Jan 24 09:22:19 UTC 2020 
 System load:  0.03              Processes:             100
 Usage of /:   10.9% of 8.68GB   Users logged in:       0
 Memory usage: 15%               IP address for enp3s5: 10.10.10.20
 Swap usage:   0%

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@vm1:~$ 

Using the console in a LXD VM

LXD has the lxc console command to give you a console to a running system container and virtual machine. You can use the console to view the boot messages as they appear, and also log in using a username and password. In the LXD profile we set up a password primarily to be able to connect through the lxc console. Let’s get a shell through the console.

$ lxc console vm1
To detach from the console, press: +a q                      [NOTE: Press Enter at this point]

Ubuntu 18.04.3 LTS vm1 ttyS0

vm1 login: ubuntu
Password: **********
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-74-generic x86_64)

* Documentation:  https://help.ubuntu.com
* Management:     https://landscape.canonical.com
* Support:        https://ubuntu.com/advantage 

System information as of Fri Jan 24 09:22:19 UTC 2020 
 System load:  0.03              Processes:             100
 Usage of /:   10.9% of 8.68GB   Users logged in:       0
 Memory usage: 15%               IP address for enp3s5: 10.10.10.20
 Swap usage:   0%

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@vm1:~$  

To exit from the console, logout from the shell first, then press Ctrl+A q.

ubuntu@vm1:~$ logout

Ubuntu 18.04.3 LTS vm1 ttyS0

vm1 login:                                               [Press Ctrl+A q]
$ 

Bonus tip: When you launch a LXD VM, you can run straight away lxc console vm1 and you get the chance to view the boot up messages of the Linux kernel in the VM as they appear.

Setting up the LXD agent inside the VM (not required anymore)

Update 22 May 2020: The following is not required at least in the Ubuntu VM images. The VM images have been updated to setup the LXD Agent automatically. When a VM is launched, it performs the installation and then it restarts once more before it becomes available.

In any VM environment the VM is separated from the host. For usability purposes, we often add a service in the VM so that it makes it easier to access the VM resources from your host. This service is available in the config device that was made available to the VM through cloud-init. At some point in the future, the LXD virtual machine images will be adapted so that they automatically setup the configuration from the config device. But for now, we do this manually by setting up the LXD agent service. First, get a shell into the virtual machine either through SSH or lxc console. We become root and perform the mount of the config device. We can see the exact files of the config device. We run ./install.sh and make the LXD Agent service run automatically in the VM. Finally, we reboot the VM so that the changes take effect.

ubuntu@vm1:~$ sudo -i
root@vm1:~# mount -t 9p config /mnt/
root@vm1:~# cd /mnt/
root@vm1:/mnt# ls -l
total 6390
-r-------- 1 999 root      745 Jan 24 09:18 agent.crt
-r-------- 1 999 root      288 Jan 24 09:18 agent.key
dr-x------ 2 999 root        5 Jan 24 09:18 cloud-init
-rwx------ 1 999 root      595 Jan 24 09:18 install.sh
-r-x------ 1 999 root 11495360 Jan 24 09:18 lxd-agent
-r-------- 1 999 root      713 Jan 24 09:18 server.crt
dr-x------ 2 999 root        4 Jan 24 09:18 systemd
root@vm1:/mnt# ./install.sh 
Created symlink /etc/systemd/system/multi-user.target.wants/lxd-agent.service → /lib/systemd/system/lxd-agent.service.
Created symlink /etc/systemd/system/multi-user.target.wants/lxd-agent-9p.service → /lib/systemd/system/lxd-agent-9p.service.

LXD agent has been installed, reboot to confirm setup.
To start it now, unmount this filesystem and run: systemctl start lxd-agent-9p lxd-agent
root@vm1:/mnt# reboot

Now the LXD Agent service is running in the VM. We are ready to use the LXD VM just like a LXD system container.

Using a LXD virtual machine

By installing the LXD agent inside the LXD VM, we can run the usual LXD commands such as lxc exec, lxc file, etc. Here is how to get a shell, either using the built-in alias lxc shell, or lxc exec to get a shell with the non-root account of the Ubuntu container images (from the repository ubuntu:).

$ lxc shell vm1
root@vm1:~# logout
$ lxc exec vm1 -- sudo --user ubuntu --login
ubuntu@vm1:~$

We can transfer files between the host and the LXD virtual machine. We create a file mytest.txt on the host. We push that file to the virtual machine vm1. The destination of the push is vm1/home/ubuntu/, where vm1 is the name of the virtual machine (or system container). It is a bit weird that we do not use : to separate the name from the path, just like in SSH and elsewhere. The reason is that : is used to specify a remote LXD server, so it cannot be used to separate the name from the path. We then perform a recursive pull of the ubuntu home directory and place it in /tmp. Finally, we have a look at the retrieved directory.

$ echo "This is a test" > mytest.txt
$ lxc file push mytest.txt vm1/home/ubuntu/
$ lxc file pull --recursive vm1/home/ubuntu/ /tmp/
$ ls -ld /tmp/ubuntu/
drwxr-xr-x 4 myusername myusername 4096 Jan  28 01:00 /tmp/ubuntu/
$ 

We can view the lxc info of the virtual machine.

$ lxc info vm1 
Name: vm1
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/01/27 20:20 UTC
Status: Stopped
Type: virtual-machine
Profiles: default, vm

Other functionality that is available to system containers should be made also available to virtual machines in the following months.

Troubleshooting

Error: unknown flag: –vm

You will get this error message when you try to launch a virtual machine while your version of LXD is 3.18 or lower. VM support has been added to LXD 3.19, therefore the version should be either 3.19 or newer.

Error: Failed to connect to lxd-agent

You can launched a LXD VM and you are trying to connect to it using lxc exec and get a shell (or run other commands). The LXD VM needs to have a service running inside the VM that will receive the lxc exec commands. This service has not been installed yet into the LXD VM, or for some reason it is not running.

Error: The LXD VM does not get automatically an IP address

The LXD virtual machine should be able to get an IP address from LXD’s dnsmasq without issues.

macvlan works as well but would not show up in lxc list vm1 until you setup the LXD Agent.

$ lxc list vm1
+------+---------+----------------------+------+-----------------+-----------+
| NAME |  STATE  |         IPV4         | IPV6 |      TYPE       | SNAPSHOTS |
+------+---------+----------------------+------+-----------------+-----------+
| vm1  | RUNNING | 192.168.1.9 (enp3s5) |      | VIRTUAL-MACHINE | 0         |
+------+---------+----------------------+------+-----------------+-----------+

I created a LXD VM and did not have to do any preparation at all!

When you lxc launchor lxc init with the aim to create a LXD VM, you need to remember to pass the --vm option in order to create a virtual machine instead of a container. To verify whether your newly created machine is a system container or a virtual machine, run lxc list and it should show you the type under the Type column.

How do I change the VM password in the LXD profile?

You can generate a new password using the following command. We are not required to echo -n in this case because mkpasswd with take care of the newline for us. We use the SHA-512 method, because this is the password hashing algorithm since Ubuntu 16.04.

$  echo "mynewpassword" | mkpasswd --method=SHA-512 --stdin
$6$BzEIxmCSyPK7$GQgw5i7SIIY0k2Oa/YmBVzmDZ4/zaxx/qJVzKBfG6uaaPYfb2efJGmJ8xxRsCaxxrYzO2NuPawrPd1DD/DsPk/
$ 

Then, run lxc profile edit vm and replace the old password field with your new one.

How do I set my public key instead of a password?

Instead of passwd, use ssh-authorized-keys. See the cloud-init example on ssh-authorized-keys.

Discussion

In LXD 3.19 there is initial support for virtual machines. As new versions of LXD are being developed, more features from system containers will get implemented into virtual machines as well. In April 2020 we will be getting LXD 4.0, long-term support for five to ten years. There is ongoing work to add as much functionality for virtual machines in order to make it into the feature freeze for LXD 4.0. If you are affected, it makes sense to follow closely the development of virtual machine support in LXD towards the LXD 4.0 feature freeze.

Permanent link to this article: https://blog.simos.info/how-to-use-virtual-machines-in-lxd/

8 comments

Skip to comment form

    • lxd_user on January 31, 2020 at 18:36
    • Reply

    Could you list all principal differences between usual container and vm container? I’ve read this article quickly, but I didn’t get them

    1. At the moment you can use lxc exec, lxc file while lxc config device is missing.

        • lxd_user on February 4, 2020 at 12:29
        • Reply

        You didn’t answer to my question. I asked about principal differences… OK, on the other hand: could you explain some reasons to use VM type of containers instead of usual system containers? Why? What benefits are there?

      1. Your questions feel like exam questions that you have been asked to answer, but instead you forward them for answering here :-).

        See my post with some background at https://blog.simos.info/comparison-between-lxc-and-lxd/

        The purpose of LXD is to be able to run Linux distributions using either Linux containers features (system containers), or hardware virtualization (virtual machine). The former requires much fewer resources than the latter. But with LXD you have the choice between the two, under the same umbrella.

        • lxd_user on February 21, 2020 at 14:57
        • Reply

        OK, I’ve read this article. Am I right that I will be able to boot few containers with seperated kernels and modules by VM containers?

      2. You can easily launch virtual machines (VMs) from the ready-made VM images that are available from the LXD image repositories.

        If you run lxc image list ubuntu:, you will be able to see the full list of VM images (column TYPE, VIRTUAL-MACHINE).

        Currently, there are VM images for Ubuntu 18.04 (lxc image list ubuntu:18.04) and newer versions. There are currently VM images for amd64, arm64, s390x and more.

        Once you boot into a virtual machine, you can switch kernel and do many other things.

        LXD users KVM and qemu and does not implement virtualization from scratch. Therefore, any guides for KVM+qemu should work here. With LXD, it just makes it so each when you find an available VM image instead of having to create it yourself.

        It should be possible soon to boot VMs with other distributions, with VM images from the images: repository. At the moment, you can run lxc image list images: | grep VIRTUAL to get the current list of VM images (there are VM images for Ubuntu, from 16.04 to current version, for amd64 and arm64).

        Note that advanced users are able to setup and run other distributions already (with some manual work). It should take a bit more so that those VM images make it into images:.

        Also, others have mentioned that they have been able to run Windows in a LXD VM. Such a feature will probably not be provided as a VM image due to Microsoft, but the facility is there because LXD VMs are KVM+qemu.

        Note that I contrast the container images with the VM images. While the repository may have a specific container image, it might not have the corresponding VM image yet.

        The terminology is that you use a container image to launch a container, and you use a virtual machine image (VM image) to launch a VM.

    • Mihai Alexandru Dumitru on May 11, 2020 at 23:26
    • Reply

    When trying to create vm profile I get:

    Error: Disk entry is missing the required “path” property

      • Mihai Alexandru Dumitru on May 13, 2020 at 04:57
      • Reply

      It seems I got an older LXD version.

Leave a Reply to Simos XenitellisCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.