How to manage LXD from within one of its containers

In a LXD (lex-dee) installation there is the host (where the LXD service is running) and there are the containers that you create. On the host, you can run the various lxc subcommands (like lxc list to get a list of the containers). The containers, on the other hand, are by default confined and isolated from the host. The containers do not access the host.

In the following we see how to poke holes in the container confinement so that some designated containers are able to manage LXD. There are some use-cases for this, in this post we see the generic functionality of getting a LXD client in a container to manage the LXD installation of its host.

There are two options to enable a container to manage the LXD installation. The first is to share the LXD Unix socket of the host to the container and the second is to use lxc remote (management over TCP/IP). In this post we see the first option.

Sharing the LXD socket to the container

On the host, the LXD client (the command line lxc utility) can access the LXD server through the LXD Unix socket. This socket in the

  • LXD deb package, is located in
    ubuntu@server:~$ ls -l /var/lib/lxd/unix.socket
    srw-rw---- 1 root lxd 0 Sep 1 2018 /var/lib/lxd/unix.socket
  • LXD snap package, is located in
    ubuntu@server:~$ ls -l /var/snap/lxd/common/lxd/unix.socket
    srw-rw---- 1 root lxd 0 Sep 1 2018 /var/snap/lxd/common/lxd/unix.socket

Therefore, if we want to enable a container to manage LXD, then we need to give access to this Unix socket to the container. In LXD 3.4 (or newer) we can share a Unix socket between the host and the container.  This is performed by using the proxy device and specifying Unix sockets.

We know from above that the unix.socket socket has permissions 0660 (rw-rw—-), owner root and group lxd.

We also know that the Ubuntu container images from ubuntu: have LXD pre-installed and is configured to get activated when a LXD client tries to access /var/lib/lxd/unix.socket. That is, if you create a ubuntu:18.04 container and then run lxc list, you have just activated the LXD service in the container! Therefore, as soon as we create the container that will manage LXD, we remove the package lxd but keep lxd-client (the LXD command-line client). We do not purge the lxd package because we want to keep as a remnant the (almost empty) directory /var/lib/lxd/. If you purged, just remember to create the directory (in the container) yourself because we are going to use it to place the host’s Unix socket in there.

Having said all that, yala, let’s go! We create the management container, then remove the lxd package (but keep the lxd-client client package).

ubuntu@server:~$ ls -l /var/snap/lxd/common/lxd/unix.socket 
srwxr-xr-x 1 root lxd 0 Sep  1 10:00 /var/snap/lxd/common/lxd/unix.socket
ubuntu@server:~$ lxc launch ubuntu:18.04 management
Creating management
Starting management
ubuntu@server:~$ lxc exec management -- sudo --user ubuntu --login
ubuntu@management:~$ sudo apt remove lxd -y
ubuntu@management:~$ exit
ubuntu@server:~$

Then, we use the LXD proxy device to link an existing Unix socket from the host into a new socket in the container.

ubuntu@server:~$ lxc config device add management lxdsocket proxy connect=unix:/var/snap/lxd/common/lxd/unix.socket listen=unix:/var/lib/lxd/unix.socket  bind=container uid=0 gid=108 mode=0660  security.uid=65534 security.gid=130
Device lxdsocket added to management

Here is again the command in a more visual way (but more difficult to copy-paste),

lxc config device add management lxdsocket proxy 
connect=unix:/var/snap/lxd/common/lxd/unix.socket
listen=unix:/var/lib/lxd/unix.socket
bind=container
uid=0
gid=108
mode=0660  
security.uid=65534
security.gid=130

Let’s dissect the long command-line from above.

  • lxc config device add, the subcommand to add a device to a container
  • management, the name of the container we created earlier
  • lxdsocket, the name of the device (arbitrary)
  • proxy, the LXD device type
  • connect=unix:/var/snap/lxd/common/lxd/unix.socket, connect to this existing Unix socket
  • listen=unix:/var/lib/lxd/unix.socket, create this Unix socket and start listening to anyone trying to connect to it,
  • bind=container, bind in the container (instead of host), which means that the connect= refers to the host and the listen= to the container. LXD will be listening to connections in the container and forward them to the specified Unix socket at the host.
  • uid=0, the User ID (UID) of the Unix socket in the container (bind=container) is 0 (root)
  • gid=108, the Group ID (GID) of the Unix socket in the container (bind=container) is 108 (lxd). An Ubuntu 18.04 (ubuntu:18.04) container sometimes defaults to the socket getting the GID 108. If unsure, verify the GID in /etc/group (in the container). Use this command, grep lxd /etc/group.
  • mode=0660, the mode of the Unix socket file (srw-rw—-)
  • security.uid=65534, the User ID of the LXD forkproxy process that runs on the host and facilitates the proxying of the Unix socket. This is the UID of the user nobody on the host.
  • security.gid=115, the Group ID of the LXD forkproxy process that runs on the host and facilitates the proxying of the Unix socket. This is the GID of lxd on the host. You need to verify this in /etc/group on the host because it is most likely different from my value. 115 was the next available group ID value at the moment when I first installed lxd on my host.

Finally, let’s try to run lxc list in the container.

ubuntu@management:~$ lxc list -c ns4
+--------------+---------+----------------------+
| NAME | STATE | IPV4 |
+--------------+---------+----------------------+
... many containers ...
+--------------+---------+----------------------+

Troubleshooting

Error “connect: no such file or directory”

Here is the full error,

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory

This means that the LXD client did not find any socket file at  /var/lib/lxd/unix.socket. Make sure that it exists.

Error “connect: connection refused”

Here is how the error looks like

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: connection refused

In this case, the LXD client found a socket file there but it is dead, it does not accept connections. Verify that the proxy device for the socket is active and running.

Error “connect: permission denied”

Here is the full error message,

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

This means that the permissions of the Unix socket were not right. Check the ownership of the Unix socket in the container (both UID and GID), and also the mode. The mode should be 0660.

If you need to remove the Unix socket from the host in order to recreate it, remove it with this command. Then, you can re-add it as needed.

ubuntu@server:~$ lxc config device remove management lxdsocket
Device lxdsocket removed from management

Permanent link to this article: https://blog.simos.info/how-to-manage-lxd-from-within-one-of-its-containers/

10 comments

1 ping

Skip to comment form

    • Bill Wood on September 12, 2018 at 03:41
    • Reply

    Cool!

  1. Thanks for this! Now I’m running LXDUI inside an LXC container, managing the host that it’s running on 🙂

  2. Remember to add yourself to the lxd group.

    • gquentin on April 10, 2019 at 15:56
    • Reply

    lxc –debug list
    DBUG[04-10|16:36:16] Connecting to a local LXD over a Unix socket
    DBUG[04-10|16:36:16] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
    Error: Get http://unix.socket/1.0: read unix @->/var/lib/lxd/unix.socket: read: connection reset by peer

    on the container:
    ls -la /var/lib/lxd/
    total 0
    drwx–x–x 1 root root 22 Mar 9 21:29 .
    drwxr-xr-x 1 root root 246 Apr 10 16:47 ..
    srw-rw—- 1 root 108 0 Apr 10 16:20 unix.socket

    on the host:
    /var/snap/lxd/common/lxd/unix.socket

    Any idea?

  3. Hi Simos. In the case when using the LXD version installed with snap inside the container (I need to have latest version, cannot use the one already available with apt, too old), how can I uninstall LXD and keep just LXD-client ?

    1. Hi Don!

      Currently, there is only a single snap package for LXD, lxd, which has both the server and the client.

      It should be feasible to create a client-only LXD snap package because you include merely the client and it does not have too many requirements as the server does. The client just communicates with the LXD server over a REST API. There are other clients as well, https://linuxcontainers.org/lxd/rest-api/ and the one used in the LXD snap package is the “Go lxd.Client” client.

      Having said that, when you install the LXD snap package, the server component is not started by default. You need to sudo lxd init in order to make it run and consume memory. In addition, if you really want to disable the server component in any case, you can additionally do so using systemctl. You do get the disadvantage in terms of space on the disk; the LXD snap package (client+server) is at 57MB (used disk space after installation). The lxc client binary is at 14MB and in total the client files could be around 20-25MB.

      1. Thanks for the details. In that case, if I don’t execute the lxd init inside the container the lxd-client should communicate with my host socket. The reason why I am asking this is because I am having problems with the socket.

        When I share my host socket with the container by setting mode to 0660, uid=1000 and gid=1000 (which is my “test” user and group inside the container) I get this (I am running commands in my container as “test” user, by using the “–user 1000 –group 1000” flags in lxc exec command):

        $ id -u
        1000
        $ id -g
        1000
        $ whoami
        test
        $ lxc list
        If this is your first time running LXD on this machine, you should also run: lxd init
        To start your first container, try: lxc launch ubuntu:18.04

        Error: Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied
        $ ls -la /var/snap/lxd/common/lxd/unix.socket
        srw-rw—- 1 root lxd 0 Sep 10 14:35 /var/snap/lxd/common/lxd/unix.socket

        So it looks like the root user is taking ownership of the socket inside the container. BUT, if I change the permissions mode to 0666 (to open r/w access to any user), then my “test” user becomes the owner of the socket:

        $ whoami
        test
        $ ls -la /var/snap/lxd/common/lxd/unix.socket
        srw-rw-rw- 1 build lxd 0 Sep 10 14:33 /var/snap/lxd/common/lxd/unix.socket
        $ exit

        So it is working with 0666 permissions, but I am not sure why changing the mode to 0666 changes the ownership of the file. Any idea on why is this happening?

      2. Sorry, pasted the output of “ls -la /var/snap/lxd/common/lxd/unix.socket” from another user, anyway I see the same issue). Here is the output when setting socket permissions mode to 0666 with user test (UID 1000, GID 1000):

        $ whoami
        test
        $ ls -la /var/snap/lxd/common/lxd/unix.socket
        srw-rw-rw- 1 test lxd 0 Sep 10 14:33 /var/snap/lxd/common/lxd/unix.socket
        $ exit

      3. When you specify the group ID in the LXD proxy device command, you should not use verbatim the group ID of the non-root account, but rather the group ID of the lxd group. Run grep lxd /etc/group on both the host and the container to find to proper values.

        When you specify the UID both times, in the first time it is 0 (root in the container) and 65534 (nobody on the host).

        I notice that your account in the container has UID 1000. Normally, the ubuntu account has UID 1000. Are you running some other Linux distribution?

        Type the full command of the LXD proxy device command so that I can have a look.

      4. I see, then I need to change my proxy device command to use the group of the lxd group.
        About the Linux distribution, I am generating my image using debootstrap to create an Ubuntu Bionic image, so that’s why my user has the UID 1000.
        This is the command that I am executing:

        lxc config device add test_container lxd_socket proxy connect=unix:/var/snap/lxd/common/lxd/unix.socket listen=unix:/var/snap/lxd/common/lxd/unix.socket bind=container mode=0660 uid=1000 gid=1000

        Looking at my /etc/group file the lxd GID is 998. Inside the container, the lxd GID is 999.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: