How to make your LXD containers get IP addresses from your LAN using macvlan

WARNING #1: By using macvlan, your computer’s network interface will appear on the network to have more than one MAC address. This is fine for Ethernet networks. However, if your interface is a Wireless interface (with security like WPA/WPA2), then the access point will ignore any other MAC addresses coming from your computer. Therefore, all these will not work in that specific case.

WARNING #2: If your host is in a virtual machine, then it is likely that the VM software will block the DHCP requests of the containers. There are options on both VMWare and Virtualbox to allow Promiscuous mode (somewhere in their Network settings). You need to enable that. Keep in mind that people reported success only on VMWare. If you make it work on VirtualBox as well, please do report below.

In LXD, you have the host and then you have the many containers on this host. The host is the computer where LXD is running. By default, all containers run hidden in a private network on the host. The containers are not accessible from the local network, nor from the Internet. However, they have network access to the Internet through the host.

How can we get some containers to receive an IP address from the LAN and be accessible on the LAN?

This can be achieved using macvlan (L2) virtual network interfaces, a feature provided by the Linux kernel.

In this post, we are going to create a new LXD profile and configure macvlan in it. Then, we launch new containers under the new profile, or attach existing containers to the new profile (so they get as well a LAN IP address).

Creating a new LXD profile for macvlan

Let’s see what LXD profiles are available.

$ lxc profile list
+------------+---------+
| NAME       | USED BY |
+------------+---------+
| default    | 11      |
+------------+---------+

There is a single profile, called default, the default profile. It is used by 11 LXD containers on this system.

We make a copy of this default profile. The new profile is called lanprofile.

$ lxc profile copy default lanprofile

$ lxc profile list
+------------+---------+
| NAME       | USED BY |
+------------+---------+
| default    | 11      |
+------------+---------+
| lanprofile | 0       |
+------------+---------+

What are the default settings of a new profile?

$ lxc profile show lanprofile
config:
 environment.TZ: ""
description: Default LXD profile
devices:
 eth0:
   nictype: bridged
   parent: lxdbr0
   type: nic
 root:
   path: /
   pool: default
   type: disk
name: lanprofile
used_by: []

We just need to change the nictype and parent to the appropriate new values and we are then ready to go. The nictype will be macvlan. The parent will be the network interface of the host. Let’s identify the correct parent, using the ip route command.

$ ip route show default 0.0.0.0/0
default via 192.168.1.1 dev enp5s12 proto static metric 100

This command shows the default network route. It also shows the name of the device (dev), which is in this case enp5s12. (Before systemd, those used to be eth0 or wlan0. Now, the name varies among computers).

Now we are ready to edit the nictype and parent fields in the lanprofile LXD profile. We use the lxc profile command to set a device parameter for the profile lanprofile. We set device.eth0.nictype to macvlan, and device.eth0.parent to enp5s12.

$ lxc profile device set lanprofile eth0 nictype macvlan

$ lxc profile device set lanprofile eth0 parent enp5s12

Here is the update lanprofile.

$ lxc profile show lanprofile
config:
  environment.TZ: ""
description: Default LXD profile
devices:
  eth0:
    nictype: macvlan
    parent: enp5s12
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: lanprofile
used_by: []

Well, that’s it. We are now ready to launch containers using this new profile, and they will get an IP address from the DHCP server of the LAN.

Launching LXD containers with the new profile

Let’s launch two containers using the new lanprofile profile and then check their IP address.

$ lxc launch -p lanprofile ubuntu:16.04 net1
Creating net1
Starting net1

$ lxc launch -p lanprofile ubuntu:16.04 net2
Creating net2
Starting net2


$ lxc exec net1 ip route
default via 192.168.1.1 dev eth0 
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.6

$ lxc exec net2 ip route
default via 192.168.1.1 dev eth0 
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.7

Both containers got their IP address from the LAN router.

Here is the router administration screen that shows the two containers. I edited the names by adding LXD in the front to make them look nicer. The containers look and feel as just like new LAN computers!

Let’s ping from one container to the other.

$ lxc exec net1 -- ping -c 3 192.168.1.7
PING 192.168.1.7 (192.168.1.7) 56(84) bytes of data.
64 bytes from 192.168.1.7: icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from 192.168.1.7: icmp_seq=2 ttl=64 time=0.067 ms
64 bytes from 192.168.1.7: icmp_seq=3 ttl=64 time=0.082 ms

--- 192.168.1.7 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2036ms
rtt min/avg/max/mdev = 0.064/0.071/0.082/0.007 ms

Let’s ping from a computer on the LAN to a container.

This is a screenshot of my phone that is pinging the net2 LXD container. Both on the same LAN.

Troubleshooting

Help! I cannot ping between the host and the containers!

To be able to get the host and containers to communicate with each other, you need some additional changes to the host in order to get added to the macvlan as well. It discusses it here, though I did not test because I do not need communication of the containers with the host. If you test it, please report below.

Help! I do not get anymore those net1.lxd, net2.lxd fancy hostnames!

The default LXD DHCP server assigns hostnames like net1.lxd, net2.lxd to each container. Then, you can get the containers to communicate with each other using the hostnames instead of the IP addresses.

When using the LAN DHCP server, you would need to configure it as well to produce nice hostnames.

Help! Can these new macvlan containers read my LAN network traffic?

The new macvlan LXD containers (that got a LAN IP address) can only see their own traffic and also any LAN broadcast packets. They cannot see the traffic meant for the host, nor the traffic for the other containers.

Permanent link to this article: https://blog.simos.info/how-to-make-your-lxd-container-get-ip-addresses-from-your-lan/

19 comments

4 pings

Skip to comment form

    • brian mullan on January 22, 2018 at 23:48
    • Reply

    Great write-up Simos…

  1. Great – thanks for the brilliant written article – it works now!

    • Byung Gon on May 2, 2018 at 07:18
    • Reply

    I did it with the above post, but the container vm can not get the ip address. Tell me how to fix the problem.
    The test environment is DHCP.

    [root@ns01 ~]# uname -a
    Linux ns01 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux (centos 7)

    [root@ns01 ~]# lxc list
    +——+———+——+——+————+———–+
    | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
    +——+———+——+——+————+———–+
    | c2 | RUNNING | | | PERSISTENT | 0 |
    +——+———+——+——+————+———–+

    [root@ns01 ~]# lxc profile show lanprofile
    config: {}
    description: Default LXD profile
    devices:
    eth0:
    name: eth0
    nictype: macvlan
    parent: ens160
    type: nic
    root:
    path: /
    pool: default
    type: disk
    name: lanprofile
    used_by:
    – /1.0/containers/c2

    [root@ns01 ~]# ifconfig ens160
    ens160: flags=4163 mtu 1500
    inet 192.168.50.30 netmask 255.255.255.0 broadcast 192.168.50.255
    inet6 fe80::9e56:786c:39be:e30a prefixlen 64 scopeid 0x20
    ether 00:0c:29:22:66:3e txqueuelen 1000 (Ethernet)
    RX packets 4342 bytes 1841714 (1.7 MiB)
    RX errors 0 dropped 12 overruns 0 frame 0
    TX packets 2515 bytes 262656 (256.5 KiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    [root@ns01 ~]# ip route show 0.0.0.0/0
    default via 192.168.50.1 dev ens160 proto static metric 100

    ======= below c2 (centos 7) ======
    [root@c2 ~]# ip -d link show eth0
    7: eth0@if2: mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/ether 00:16:3e:f4:6f:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    macvlan mode bridge addrgenmode eui64

    Thanks.

    1. Do you use a VM for the host? If so, then the VM might be the issue.
      Also, if the host is on a wireless interface, then that’s definitely the issue.

    • Ray Jender on May 26, 2018 at 14:47
    • Reply

    I am getting this error when I try to start a macvlan container :

    ray@USN-LPC:/var/lib/lxd/containers$ lxc start LPC2
    error: Failed to run: /usr/bin/lxd forkstart LPC2 /var/lib/lxd/containers /var/log/lxd/LPC2/lxc.conf:
    Try `lxc info –show-log LPC2` for more info

    ray@USN-LPC:/var/lib/lxd/containers$ lxc info LPC2
    Name: LPC2
    Remote: unix://
    Architecture: x86_64
    Created: 2018/05/26 13:14 UTC
    Status: Stopped
    Type: persistent
    Profiles: lanprofile

    ray@USN-LPC:/var/lib/lxd/containers$ lxc profile show lanprofile
    config:
    environment.http_proxy: “”
    user.network_mode: “”
    description: Default LXD profile
    devices:
    eth0:
    nictype: macvlan
    parent: enp0s3
    type: nic
    root:
    path: /
    pool: lxdpool
    type: disk
    name: lanprofile
    used_by:
    – /1.0/containers/LPC2

    Ideas?

    Thanks.

    Ray

    1. What does *lxc info –show-log LPC2* show? It should give a hint of the exact error.

  2. This worked perfectly on a CentOS 7.5 1804 (core) host on 8/21/2018. I created an LXD ZFS-backed container for Ubuntu 18.04 exactly as you describe, and it automatically got an address assigned on my home network LAN by the DHCP server on that LAN. The private network on the host that LXD gave me was useless, but the methods you illustrate here are very useful. Thanks for your work, and for the clarity of exposition as well! Bravo!

    1. Thanks Robert for the kind words!

    • Wouter on September 11, 2018 at 18:34
    • Reply

    Thanks for your article. I did not succeed with macvlan. On Ubuntu 17.10 server I have a container running with the name proxy:

    root@box:~# lxc list
    +——-+———+——+——+————+———–+
    | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
    +——-+———+——+——+————+———–+
    | proxy | RUNNING | | | PERSISTENT | 0 |

    with this MAC:

    root@box:~# lxc exec proxy ifconfig |grep ether
    ether 00:16:3e:23:71:a9 txqueuelen 1000 (Ethernet)

    that should be getting an IP; at least, the container is asking for one and is offered one:
    Sep 11 12:06:51 box dhcpd[5314]: DHCPDISCOVER from 00:16:3e:23:71:a9 via enp1s0f1
    Sep 11 12:06:51 box dhcpd[5314]: DHCPOFFER on 192.168.12.25 to 00:16:3e:23:71:a9 via enp1s0f1

    but is not:

    root@box:/var/log# lxc exec proxy ip a |grep inet\ 192
    root@box:/var/log# lxc exec proxy ifconfig |grep 192

    and ICMP doesnt work:
    root@box:/var/log# ping 192.168.12.25 -c1
    PING 192.168.12.25 (192.168.12.25) 56(84) bytes of data.
    From 192.168.12.1 icmp_seq=1 Destination Host Unreachable

    root@box:/var/log# lxc profile show lanprofile |grep “nictype|parent”
    nictype: macvlan
    parent: enp1s0f1

    I stopped and started the container several times and also did a reinstall / reinit of lxd several times. Please help :).

    root@box:/var/log# lxc –version
    2.18

    root@box:/var/log# lxc network list |grep “lx|enp1s0f1”
    | enp1s0f1 | physical | NO | | 1 |
    | lxcbr0 | bridge | NO | | 0 |
    | lxdbr0 | bridge | YES | | 0 |

    1. Since your container requests an IP address and is offered one, then any problem should be related to the operating system of the container. I cannot think of the scenario that the DHCPOFFER being somehow blocked and not reaching the container. You can get the DHCP client in the container to output debugging information in order to see how it processes the DHCPOFFER.

      In addition, I notice that you are running LXD 2.18. If I remember correctly, you probably use a PPA or the backports repository. I suggest to upgrade to the snap package, which currently has LXD 3.4. An upgrade should not directly fix the issue that you are facing. However, version 2.18 of LXD is not supported as far as I remember. Only version 2.0.11 (Ubuntu 16.04), 3.0.x (Ubuntu 18.04) are supported until the EOL of the corresponding LTS Ubuntu version.

        • Wouter on September 12, 2018 at 08:38

        I was runnng 2.18 and upgraded via backports (thanks for the hint) to 2.21. No luck. Also, I launched both 16.04 and 18.04 containers with 2.21: no luck. dhclient -v eth0 says
        Listening on LPF/eth0/00:16:3e:db:e2:ce
        Sending on LPF/eth0/00:16:3e:db:e2:ce
        Sending on Socket/fallback
        DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x548c1235)
        how… I don’t understand, but it seems the answer (DHCPOFFER) is not received. Giving up for now. First, find time to reinstall the box with Ubuntu 18.04 and then further with LXD 3.x.

        thanks

      1. If there was virtualization in play (like VirtualBox), then this would be explained easily. There is an issue with Virtualbox and the result is exactly what you describe. That is, the container does not receive the DHCPOFFER in order to continue with the rest of the protocol. The workaround in Virtualbox would be to put the host’s interface in promiscuous mode.

    • Wouter on September 12, 2018 at 17:03
    • Reply

    @Simos Xenitellis on September 12, 2018 at 16:24
    thanks for thinking with me. Its Ubuntu 17.10 on bare metal. Also, I installed the snap package with lxd 3.x (couldnt wait for finding time to install 18.04 :)) and it’s the same. Both 16.04 and 18.04 containers request DHCP, an offer is sent but not received.

    1. The macvlan functionality works in LXD, therefore I assume there is some issue with the network driver or network settings.

      I do not have the full picture. From the interface name (enp1s0f1), I assume it’s the second network interface of two? Can you verify with tshark that the DHCPOFFER is sent to the correct network interface?

        • Wouter on September 13, 2018 at 08:44

        Yes, enp1s0f1 is the second part of this NIC
        https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf
        Yes, I did tcpdump on enp1s0f1 and saw the DHCPOFFER there; it’s also the only interface where dhcpd is listening on.

      1. I think that this stage you can post this issue on https://discuss.linuxcontainers.org/
        If it is a bug in LXD, it should be then reported on Github so that it gets fixed.

    • gunnar on October 5, 2018 at 05:18
    • Reply

    I can confirm that this is not working on a VirtualBox VM (ubuntu 18.04) as LXD host.

      • gunnar on October 5, 2018 at 05:26
      • Reply

      side comment … if u have a proxmox hist inside a VirtualBox VM there is no problem whatsoever to get public IP’s for containers working (shared adapter, no promiscous mode). So whether anyone could break down how proxmox handles networking for LXC containers cure might be around the corner.

    1. I have tried with Virtualbox (LXD running on Ubuntu 18.04 in a VirtualBox VM).
      I set up the Promiscuous setting in Virtualbox but macvlan did not work either.

      But when you set as well the host’s Ethernet interface in PROMISC mode, then it works.
      You can set it to PROMISC mode

      ip link set eth5 promisc on

      or

      ifconfig eth5 promisc

      When you run proxmox, is the interface in PROMISC mode?

  1. […] How to make your LXD containers get IP addresses from your LAN using macvlan […]

  2. […] of using a bridge, I will try to expose the containers directly on the LAN with macvlan. This guide is helpful. I change the default profile to use […]

  3. […] and the networking. The default LXD profile is suitable for this. You may use a bridge profile or macvlan profile […]

  4. […] post I used for setting up macvlan for the containers linked to this post, which explains how to have […]

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: