How to use LXD container hostnames on the host in Ubuntu 18.04

Update: 12 Oct 2018The systemd service file and the LXD lxdbr0 configuration have been updated according to the comment by mDfRgmd. Please check again and verify that you are using the updated version.

If you have two LXD containers, mycontainer1 and mycontainer2, then you can reference each other with those handy *.lxd hostnames like this,

$ lxc exec mycontainer1 -- sudo --user ubuntu --login
ubuntu@mycontainer1:~$ ping mycontainer2.lxd
PING mycontainer2.lxd(mycontainer2.lxd (fd42:cba6:557e:1a5a:24e:3eff:fce2:8d3)) 56 data bytes
64 bytes from mycontainer2.lxd (fd42:cba6:557e:1a5a:24e:3eff:fce2:8d3): icmp_seq=1 ttl=64 time=0.125 ms
^C
--- mycontainer2.lxd ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.125/0.125/0.125/0.000 ms
ubuntu@mycontainer1:~$

Those hostnames are provided automatically by LXD when you use a default private bridge like lxdbr0. They are provided by the dnsmasq service that LXD starts for you, and it’s a service that binds specifically on that lxdbr0 network interface.

LXD does not make changes to the networking of the host, therefore you cannot use those hostnames from your host,

ubuntu@mycontainer1:~$ exit
$ ping mycontainer2.lxd
ping: unknown host mycontainer2.lxd
Exit 2

In this post we are going to see how to set up the host on Ubuntu 18.04 (any Linux distribution that uses systemd-resolve) so that the host can access the container hostnames.

The default configuration per systemd of the lxdbr0 bridge on the host is

$ systemd-resolve --status lxdbr0
Link 2 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

The goal is to add the appropriate DNS server entries to appear in that configuration.

Let’s get first the IP address of LXD’s dnsmasq server for the network interface lxdbr0.

$ ip addr show dev lxdbr0
2: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fe:2b:da:d9:49:4a brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:6a89:42d0:60b::1/64 scope global 
valid_lft forever preferred_lft forever
inet6 fe80::10cf:51ff:fe05:5383/64 scope link 
valid_lft forever preferred_lft forever

The IP address of the lxdbr0 interface in this case is 10.10.10.1 and that is the IP of LXD’s DNS server.

Now we can move on by configuring the host to consult LXD’s DNS server. In the following, you need to make the additions to the lxdbr0 bridge interface, and then select whether to perform the temporary or permanent systemd network configuration.

Additions to the lxdbr0 bridge interface (new!)

The following additions to the lxdbr0 bridge interface are required so that you avoid a DNS loop. A DNS loop may happen if there are two DNS servers and they end up asking each other for a name resolution. The following additions avoid a potential DNS loop.

First, let’s verify the name of the network interface. It is lxdbr0, and it’s a managed network interface.

$ lxc network list
+-----------+----------+---------+-------------+---------+
| NAME | TYPE | MANAGED | DESCRIPTION | USED BY |
+-----------+----------+---------+-------------+---------+
| lxdbr0 | bridge | YES | | 15 |
+-----------+----------+---------+-------------+---------+
..

What is the existing (and default) configuration of the network interface lxdbr0?

$ lxc network show lxdbr0
config:
ipv4.address: 10.100.1.1/24
ipv4.nat: "true"
ipv6.address: fd42:d282:a2d:f60a::1/64
ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by: []
managed: true
status: Created
...snip...

We need to add in there specific instructions to dnsmasq (the DNS server that is used in LXD for the managed network interfaces) so that dnsmasq

  1. knows it is the authoritative DNS Zone for the .lxd domain (auth-zone=lxd)
  2. adds protection against loops (dns-loop-detect)
    –dns-loop-detect,
    Enable code to detect DNS forwarding loops; ie the situation where a query sent to one of the upstream server eventually returns as a new query to the dnsmasq instance. The process works by generating TXT queries of the form .test and sending them to each upstream server. The hex is a UID which encodes the instance of dnsmasq sending the query and the upstream server to which it was sent. If the query returns to the server which sent it, then the upstream server through which it was sent is disabled and this event is logged. Each time the set of upstream servers changes, the test is re-run on all of them, including ones which were previously disabled.

You can add these two options with the following command.

$ echo -e "auth-zone=lxd\ndns-loop-detect" | lxc network set lxdbr0 raw.dnsmasq -

Let’s verify the change.

$ lxc network show lxdbr0
config:
ipv4.address: 10.100.1.1/24
ipv4.nat: "true"
ipv6.address: fd42:d282:a2d:f60a::1/64
ipv6.nat: "true"
raw.dnsmasq: |
auth-zone=lxd
dns-loop-detect
description: ""
name: lxdbr0
type: bridge
used_by: []
managed: true
status: Created
...

We have now edited the LXD managed network interface and are ready to configure systemd-resolved to resolve those .lxd domain names.

Temporary systemd network configuration

Run the following command to configure temporarily the interface and add the DNS service details.

$ sudo systemd-resolve --interface lxdbr0 --set-dns 10.10.10.1 --set-domain lxd

In this command,

  1. we specify the network interface lxdbr0
  2. we set the DNS server to the IP address of the lxdbr0, the interface that dnsmasq is listening on.
  3. we set the domain to lxd, as the hostnames are of the form mycontainer.lxd.

Now, the configuration looks like

$ systemd-resolve --status lxdbr0
Link 2 (lxdbr0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.10.10.1
          DNS Domain: lxd

You can now verify that you can, for example, get the IP address of the container by name:

$ host mycontainer1.lxd
mycontainer.lxd has address 10.10.10.88
mycontainer.lxd has IPv6 address fd42:8196:99f3:52ad:216:3eff:fe0f:bacb
$

Note: The first time that you try to resolve such a hostname, it will take a few seconds for systemd-resolved to complete the resolution. You will get the result shown above, but the command will not return immediately. The reason is that systemd-resolved is waiting to get a resolution from your default host’s DNS server, and you are waiting for that resolution to timeout. The next attempts will be cached and return immediately.

You can also revert these settings with the following command,

$ systemd-resolve --interface lxdbr0 --revert
$ systemd-resolve --status lxdbr0
Link 3 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
$

In general, this is a temporary network configuration and nothing has been saved to a file. When we reboot the computer, the configuration is gone.

Permanent systemd network configuration (updated)

We are going to set up systemd to run automatically the temporary network configuration whenever LXD starts. That is, as soon as lxdbr0 is up, our additional script will run and configure the per-link network.

First, create the following auxiliary script files.

$ cat /usr/local/bin/lxdhostdns_start.sh 
#!/bin/sh

LXDINTERFACE=lxdbr0
LXDDOMAIN=lxd
LXDDNSIP=`ip addr show lxdbr0 | grep -Po 'inet \K[\d.]+'`

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} \
                         --set-dns ${LXDDNSIP} \
                         --set-domain ${LXDDOMAIN}

$ cat /usr/local/bin/lxdhostdns_stop.sh 
#!/bin/sh

LXDINTERFACE=lxdbr0

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} --revert

Second, make them executable.

$ sudo chmod +x /usr/local/bin/lxdhostdns_start.sh /usr/local/bin/lxdhostdns_stop.sh

Third, create the following systemd service file. NOTE: the following has been updated (12 Oct 2018) according to the comment by mDfRgmd. The two changes are shown in bold/italic.

$ sudo cat /lib/systemd/system/lxd-host-dns.service 
[Unit]
Description=LXD host DNS service
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/local/bin/lxdhostdns_start.sh
RemainAfterExit=true
ExecStop=/usr/local/bin/lxdhostdns_stop.sh
StandardOutput=journal

[Install]
WantedBy=multi-user.target

This file

  • will activate after the lxd-containers.service service (therefore, lxdbr0 is up).
  • it is a oneshot (runs until completion before the next service).
  • it runs the respective scripts on ExecStart and ExecStop.
  • the RemainAfterExit is true, which means that it appears as running in systemd.
  • if something is wrong, it will be reported in the journal.
  • it gets installed in the multi-user target (same as the LXD service).

Fourth, now we reload systemd and enable the new service. The service is enabled so that when we reboot, it will start automatically.

$ sudo systemctl daemon-reload $ sudo systemctl enable lxd-host-dns.service Created symlink /etc/systemd/system/multi-user.target.wants/lxd-host-dns.service → /lib/systemd/system/lxd-host-dns.service. $

Note: This should work better than the old (next section) instructions. Those old instructions would fail if the lxdbr0 network interface was not up. Still, I am not completely happy with this new section. It appears that when you explicitly start or stop the new service, the action may not run. To be tested.

Troubleshooting

The first name resolution for each container takes long!

Indeed, when we try to resolve a name such as mycontainer.lxd, it takes some time for the resolution to complete. The reason is that systemd-resolved waits for the default DNS server (which has no knowledge of the *.lxd names) to try to resolve the name. The wait is the timeout time for the default DNS server to resolve the name. Ideally, systemd-resolved should understand that we used the .lxd suffix to the hostname, do a match with the Domains field in the per-link configuration, and only query the per-link DNS server.

It is not clear how to configure systemd to do just that, to not attempt to query the default DNS server for names that already have the .lxd suffix. If you have figure out a way for this, please write below.

Note though that subsequent queries for the same name return immediately with the correct answer because the answer is cached.

(old section, not working) Permanent network configuration

In systemd, we can add per network interface configuration by adding a file in /etc/systemd/network/.

It should be a file with the extension .network, and the appropriate content.

Add the following file

$ cat /etc/systemd/network/lxd.network 
[Match]
Name=lxdbr0

[Network]
DNS=10.100.100.1
Domains=lxd

We chose the name lxd.network for the filename. As long as it has the .network extension, we are fine.

The [Match] section matches the name of the network interface, which is lxdbr0. The rest will only apply if the network interface is indeed lxdbr0.

The [Network] section has the specific network settings. We set the DNS to the IP of the LXD DNS server. And the Domains to the domain suffix of the hostnames. The lxd in Domains is the suffix that is configured in LXD’s DNS server.

Now, let’s restart the host and check the network configuration.

$ systemd-resolve --status
...
Link 2 (lxdbr0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.100.100.1
                      fe80::a405:eade:4376:3817
          DNS Domain: lxd

Everything looks fine. By doing the configuration this way, systemd-resolve also picked up automatically the IPv6 address.

Conclusion

We have seen how to setup the host on a LXD installation so that processes on the host are able to see the hostnames of the containers. For Ubuntu 18.04 or any distribution that uses systemd for the DNS client needs.

If you use Ubuntu 16.04, then it requires a different way involving the dnsmasq-base configuration. There are instructions on this on the Internet, ask if you cannot find them.

Permanent link to this article: https://blog.simos.info/how-to-use-lxd-container-hostnames-on-the-host-in-ubuntu-18-04/

5 comments

Skip to comment form

    • Rudi Servo on September 25, 2018 at 15:36
    • Reply

    On ubuntu 18.04 resolved stays at 100% cpu usage causo of DNSMASQ
    Adding this DNSMASQ_EXCEPT=lo to /etc/default/dnsmasq will solve the problem has stated in

    https://askubuntu.com/questions/909591/systemd-resolve-high-cpu-usage-after-update-to-17-04/968309

      • Rudi Servo on September 25, 2018 at 15:59
      • Reply

      sorry it didnt work has expected still debugging :/

    1. The discussion has moved to this thread, https://discuss.linuxcontainers.org/t/ubuntu-18-04-infinite-loop-dnsmasq-systemd-resolved/2768/

      Thanks for narrowing it down to an issue with IPv6. Specifically, if IPv6 is not (fully?) available, then there are excessive requests with lead to high CPU usage.

  1. The /lib/systemd/system/lxd-host-dns.service syntax didn’t work for me under Ubuntu 18.04. Here is what worked:

    [Unit]
    Description=LXD host DNS service
    After=multi-user.target

    [Service]
    Type=simple
    ExecStart=/usr/local/bin/lxdhostdns_start.sh
    RemainAfterExit=true
    ExecStop=/usr/local/bin/lxdhostdns_stop.sh
    StandardOutput=journal

    [Install]
    WantedBy=multi-user.target

    Notice the Type=simple and After= change.

    Thank you for your blog. There are little information about LXD and what you provide is very informative.

    1. Many thanks mDfRgmd!

      I have updated the post to include
      1. the additions in the LXD managed network interface (auth-zone=lxd, detect-dns-loop)
      2. the changes the systemd service file.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: