How to manage LXD from within one of its containers

In a LXD (lex-dee) installation there is the host (where the LXD service is running) and there are the containers that you create. On the host, you can run the various lxc subcommands (like lxc list to get a list of the containers). The containers, on the other hand, are by default confined and isolated from the host. The containers do not access the host.

In the following we see how to poke holes in the container confinement so that some designated containers are able to manage LXD, the one installed on the host. There are some use-cases for this, in this post we see the generic functionality of getting a LXD client in a container to manage the LXD installation of its host.

There are two options to enable a container to manage the LXD installation. The first is to share the LXD Unix socket of the host to the container and the second is to use lxc remote (management over TCP/IP). In this post we see the first option.

Sharing the LXD socket to the container

On the host, the LXD client (the command line lxc utility) can access the LXD server through the LXD Unix socket. This socket in the

  • LXD deb package, is located in
    ubuntu@server:~$ ls -l /var/lib/lxd/unix.socket

    srw-rw---- 1 root lxd 0 Sep 1 2018 /var/lib/lxd/unix.socket
  • LXD snap package, is located in
    ubuntu@server:~$ ls -l /var/snap/lxd/common/lxd/unix.socket

    srw-rw---- 1 root lxd 0 Sep 1 2018 /var/snap/lxd/common/lxd/unix.socket

Therefore, if we want to enable a container to manage LXD, then we need to give access to this Unix socket to the container. Since LXD 3.4 we can now share a Unix socket between the host and the container. This is performed by using the LXD proxy device and specifying Unix sockets.

We know from above that the unix.socket socket has permissions 0660 (rw-rw—-), owner root and group lxd.

We also know that the Ubuntu container images from ubuntu: have LXD pre-installed and is configured to get activated when a LXD client tries to access /var/lib/lxd/unix.socket. That is, if you create a ubuntu:18.04 container and then run lxc list, you have just activated the LXD service in the container! Therefore, as soon as we create the container that will manage LXD, we remove the package lxd but keep lxd-client (the LXD command-line client). We do not purge the lxd package because we want to keep as a remnant the (almost empty) directory /var/lib/lxd/. If you purged, just remember to create the directory (in the container) yourself because we are going to use it to place the host’s Unix socket in there.

Having said all that, yala, let’s go! We create the management container, then remove the lxd package (but keep the lxd-client client package).

ubuntu@server:~$ ls -l /var/snap/lxd/common/lxd/unix.socket 
srwxr-xr-x 1 root lxd 0 Sep  1 10:00 /var/snap/lxd/common/lxd/unix.socket
ubuntu@server:~$ lxc launch ubuntu:18.04 management
Creating management
Starting management
ubuntu@server:~$ lxc exec management -- sudo --user ubuntu --login
ubuntu@management:~$ sudo apt remove lxd -y
ubuntu@management:~$ exit
ubuntu@server:~$

Then, we use the LXD proxy device to link an existing Unix socket from the host into a new socket in the container.

ubuntu@server:~$ lxc config device add management lxdsocket proxy connect=unix:/var/snap/lxd/common/lxd/unix.socket listen=unix:/var/lib/lxd/unix.socket  bind=container uid=0 gid=108 mode=0660  security.uid=65534 security.gid=130
Device lxdsocket added to management

Here is again the command in a more visual way (but more difficult to copy-paste),

lxc config device add management lxdsocket proxy 
connect=unix:/var/snap/lxd/common/lxd/unix.socket
listen=unix:/var/lib/lxd/unix.socket
bind=container
uid=0
gid=108
mode=0660  
security.uid=65534
security.gid=130

Let’s dissect the long command-line from above.

  • lxc config device add, the subcommand to add a device to a container
  • management, the name of the container we created earlier
  • lxdsocket, the name of the device (arbitrary)
  • proxy, the LXD device type
  • connect=unix:/var/snap/lxd/common/lxd/unix.socket, connect to this existing Unix socket
  • listen=unix:/var/lib/lxd/unix.socket, create this Unix socket and start listening to anyone trying to connect to it,
  • bind=container, bind in the container (instead of host), which means that the connect= refers to the host and the listen= to the container. LXD will be listening to connections in the container and forward them to the specified Unix socket at the host.
  • uid=0, the User ID (UID) of the Unix socket in the container (bind=container) is 0 (root)
  • gid=108, the Group ID (GID) of the Unix socket in the container (bind=container) is 108 (lxd). An Ubuntu 18.04 (ubuntu:18.04) container sometimes defaults to the socket getting the GID 108. If unsure, verify the GID in /etc/group (in the container). Use this command, grep lxd /etc/group.
  • mode=0660, the mode of the Unix socket file (srw-rw—-)
  • security.uid=65534, the User ID of the LXD forkproxy process that runs on the host and facilitates the proxying of the Unix socket. This is the UID of the user nobody on the host.
  • security.gid=115, the Group ID of the LXD forkproxy process that runs on the host and facilitates the proxying of the Unix socket. This is the GID of lxd on the host. You need to verify this in /etc/group on the host because it is most likely different from my value. 115 was the next available group ID value at the moment when I first installed lxd on my host.

Finally, let’s try to run lxc list in the container.

ubuntu@management:~$ lxc list -c ns4
+--------------+---------+----------------------+
| NAME | STATE | IPV4 |
+--------------+---------+----------------------+
... many containers ...
+--------------+---------+----------------------+

Troubleshooting

Error “connect: no such file or directory”

Here is the full error,

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory

This means that the LXD client did not find any socket file at  /var/lib/lxd/unix.socket. Make sure that it exists.

Error “connect: connection refused”

Here is how the error looks like

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: connection refused

In this case, the LXD client found a socket file there but it is dead, it does not accept connections. Verify that the proxy device for the socket is active and running.

Error “connect: permission denied”

Here is the full error message,

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

This means that the permissions of the Unix socket were not right. Check the ownership of the Unix socket in the container (both UID and GID), and also the mode. The mode should be 0660.

If you need to remove the Unix socket from the host in order to recreate it, remove it with this command. Then, you can re-add it as needed.

ubuntu@server:~$ lxc config device remove management lxdsocket
Device lxdsocket removed from management

Permanent link to this article: https://blog.simos.info/how-to-manage-lxd-from-within-one-of-its-containers/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.