How to use the LXD Proxy Device to map ports between the host and the containers

LXD supports proxy devices, which is a way to proxy connections between the host and containers. This includes TCP, UDP and Unix socket connections, in any combination between each other, in any direction. For example, when someone connects to your host on port 80 (http), then this connection can be proxied to a container using a proxy device. In that way, you can isolate your Web server into a LXD container. By using a TCP proxy device, you do not need to use iptables instead.

There are 3³=9 combinations for connections between TCP, UDP and Unix sockets, as follows. Yes, you can proxy, for example, a TCP connection to a Unix socket!

  1. TCP to TCP, for example, to expose a container’s service to the Internet.
  2. TCP to UDP
  3. TCP to Unix socket
  4. UDP to UDP
  5. UDP to TCP
  6. UDP to Unix socket
  7. Unix socket to Unix socket, for example, to share the host’s X11 socket to a container. Or, to make available a host’s Unix socket into the container.
  8. Unix socket to TCP
  9. Unix socket to UDP

Earlier I wrote that you can make a connection in any direction. For example, you can expose the host’s Unix socket for X11 into the container so that the container can run X11 applications and have them appear on the host’s X11 server. Or, in the other way round, you can make available LXD’s Unix socket at the host to a container so that you can manage LXD from inside a container.

Note that LXD 3.0.x only supports TCP to TCP proxy devices. Support for UDP and Unix sockets was added in later versions.

Launching a container and setting up a Web server

Let’s launch a container, install a Web server, and, then expose the Web server to the local network (or the Internet, if you are using a VPS/Internet server).

First, launch the container.

$ lxc launch ubuntu:18.04 mycontainer
Creating mycontainer
Starting mycontainer

We get a shell into the container, update the package list and install nginx. Finally, verify that nginx is running.

ubuntu@mycontainer:~$ sudo apt update
ubuntu@mycontainer:~$ sudo apt install -y nginx
ubuntu@mycontainer:~$ curl http://localhost
...
 Welcome to nginx! 

Exposing the Web server of a container to the Internet

We logout to the host and verify that there is no Web server already running on port 80. If port 80 is not available on your host, change it to something else, like 8000. Finally, we create the TCP to TCP LXD Proxy Device.

ubuntu@mycontainer:~$ logout
$ lxc config device add mycontainer myport80 proxy listen=tcp:0.0.0.0:80 connect=tcp:127.0.0.1:80
Device myport80 added to mycontainer

The command that creates the proxy device is made of the following components.

  1. lxc config device add, we configure to have a device added,
  2. mycontainer, to the container mycontainer,
  3. myport80, with name myport80,
  4. proxy, a proxy device, we are adding a LXD Proxy Device.
  5. listen=tcp:0.0.0.0:80, we listen (on the host by default) on all network interfaces on TCP port 80.
  6. connect=tcp:127.0.0.1:80, we connect (to the container by default) to the existing TCP port 80 on localhost, which is our nginx.

Note that previously you would specify hostnames when you were creating LXD Proxy Devices. This is no longer supported (has security implications), therefore you get an error if you specify a hostname such as localhost. This post was primarily written because the top Google result on proxy devices is an old Reddit read-only post that suggests to use localhost.

Let’s test that the Web server in the container is accessible on the host. We can use both localhost (or 127.0.0.1) on the host to access the website of the container. We can also use the public IP address of the host (in this case, the LAN IP address) to access the container.

$ curl http://localhost
...
 Welcome to nginx! 
...
$ curl http://192.168.1.100
...
 Welcome to nginx! 
...

Other features of the proxy devices

By default, a proxy device exposes an existing service in the container to the host. If we need to expose an existing service on the host to a container, we would add the parameter bind=container to the proxy device command.

You can expose a single webserver to a port on the host. But how do you expose many web servers in containers to the host? You can use a reverse proxy that goes in front of the containers. To retain the remote IP address of the clients visiting the Web servers, you can add the proxy_protocol=true to enable support for the PROXY protocol. Note that you also need to enable the PROXY protocol on the reverse proxy.

Permanent link to this article: https://blog.simos.info/how-to-use-the-lxd-proxy-device-to-map-ports-between-the-host-and-the-containers/

13 comments

Skip to comment form

    • Yoann on September 7, 2019 at 09:06
    • Reply

    Hi, thanks for your post about lxc/d.
    Proxy device seems great, but in the use case you describe there is a problem, because in the access log of ngix container every requests seems to came from 127.0.0.1 instead of the real IP of the client.
    Yoann

    1. Seeing 127.0.0.1 in the logs instead of the real IP of the client is a typical issue with any kind of proxies.

      To bypass this issue, you use the PROXY protocol,
      https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

      In practice, in LXD, you add the proxy_protocol=true when you create the proxy device. That is what is only required from LXD (LXD 3.16 or newer).

      On the receiving end (your Web server), you need to activate support for the PROXY protocol.
      Depending on your Web server, there are instructions on how to do this.
      If you use either nginx or Apache, have a look at https://www.linode.com/docs/applications/containers/beginners-guide-to-lxd-reverse-proxy/
      Note that in your case, the trusted IP address is “127.0.0.1”, to be used in your configuration on the Web server.

        • Yoann on September 11, 2019 at 17:26
        • Reply

        THANKS A LOT, it was the missing piece of my configuration (multiple LXD container for different versions of php/mysql/postgresql).
        I’m really HAPPY.

  1. Hi Simos,

    compared to use traditional iptables NAT, what’s the benefits of proxy?

    Regards,

    1. If you use iptables, you would need to

      1. figure out the actual full iptables command which is not trivial
      2. make this command persistent so that when you reboot, the rule will still be applied. This means learning to use iptables-persistent.
      3. take care of the case of a firewall like ufw. When you apply firewall rules, they may clear your own iptables.
      4. set a static IP address to the container because if the IP changes, the iptables rule needs updating.

      The LXD proxy does not have these issues. You do not specify the IP address of the container because LXD already knows it. When you start the container, the LXD proxy is automatically set up.

      The benefit of iptables is that of performance. iptables creates a rule for the Linux kernel to take care off, while the LXD proxy is a process that provides the service. You would consider such benefit if you are running a very fast network. In other words, if you where to consider the issue of performance, you would have already created a benchmark to compare the two and found the sweet spot of traffic that the LXD proxy can sufficiently serve. If you actually do this, please report back! 🙂

    • daniele on May 11, 2020 at 15:56
    • Reply

    In cluster mode (lxd 3 or 4) is possible to expose the container port to all the cluster nodes. So I do not need to know in which host the container is running. Example: container expose port 22 as port 2222, and I have 2 cluster nodes node1 and node2, I can connect either with ssh node1 -p 2222 or ssh node2 -p 2222 to the same container. Thanks

    1. In a cluster, a container is running on only one of the hosts. Therefore, it is not possible to expose the container port to the other hosts of the cluster.
      What you can do though, is setup a reverse proxy on the rest of the hosts of the cluster, and have that reverse proxy direct the traffic to the container in question.

    • Scott Lair on June 6, 2020 at 17:36
    • Reply

    I am using debian as my distro of choice which does not yet support lxd natively. Do you know if similar functionality exists in lxc containers?
    thanks.

    1. Hi!

      I suppose you mean lxc as in LXC in https://blog.simos.info/comparison-between-lxc-and-lxd/
      LXC does not have proxy devices, which means you have to use iptables/nftables yourself.
      See the package iptables-persistent for persistence of your rules.

      You can install the snap package of LXD in Debian. According to https://snapcraft.io/lxd (see Stats section), LXD is the most popular distribution after Ubuntu.

        • Scott Lair on June 9, 2020 at 17:57
        • Reply

        Hi Simos,

        Yes, I do mean LXC. I was trying to setup a legacy printer in a debian jessie LXC container, but am having trouble and thought maybe a proxy would help. I also tried an LXD container, but I am unable to get cups to install. Ah well, I’ll keep trying.

        thanks

        Scott

      1. Hi Scott,

        I assume that this legacy printer only comes with some obscure .deb package that can be installed in some very old version of Ubuntu?

        Here is what I suggest:

        1. Select the appropriate distribution, such as Ubuntu 12.04 or Ubuntu 14.04.
        2. Follow my X2Go tutorial so that you can get a desktop from that old Ubuntu version.
        3. Use LXD’s USB passthrough support to make the printer appear in that container. Install the drivers.
        4. Once you get the printer working, you can either print from within that container, or just share the printer to your local desktop.
    • johncausing on August 9, 2020 at 12:25
    • Reply

    Hi Simos,

    Thank you so much for all of your articles. I really love it and helped me a lot.

    Using this tutorial I was able to implement this to connect to my multiple LXD containers using SSH. I can connect from my local computer to those LXD containers. So this is perfect

    But using this tutorial, if I setup multiple containers to access the nginx from the internet, how can I make this work if I have 5 LXD’s running the same nginx with port 80?

    I following this tutorial and this is working just fine: https://www.digitalocean.com/community/tutorials/how-to-host-multiple-web-sites-with-nginx-and-haproxy-using-lxd-on-ubuntu-16-04

    This domain: http://lxd1.causingdesigns.net/

    That is currently connected to one of my LXC using hafproxy.

    How can I connect to nginx from the internet using proxy device?

    1. Thanks John!

      When you need to use SSH to connect to each individual LXD containers, you configure to use separate ports for SSH, such as 22122, 22222, 22322, 22422 and so on. This is because there is no easy way for your server to know from before which server you intended to connect to. There are some workarounds with bastion hosts, but it’s simpler with separate ports. I suppose you use separate ports as well?

      In the case of the HTTP/HTTPS protocol, the protocol allows to peek inside the packets and figure out which website was intended to connect to. The software that does this task, is the reverse proxy, such as HAProxy and even nginx in reverse-proxy mode.

      My DigitalOcean tutorial shows how to configure HAProxy to perform this reverse-proxy task. That is, you configure the host to direct any connections for ports 80 and 443 to the reverse-proxy container, then inside the reverse-proxy container (HAProxy, nginx or other) you configure to identify which website is supposed to connect to, and redirect to the appropriate container.

      Perhaps nginx as a reverse-proxy is easier to use. Have a look at this tutorial, https://www.linode.com/docs/applications/containers/beginners-guide-to-lxd-reverse-proxy/

Leave a Reply to Yoann Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: