How to get LXD containers obtain IP from the LAN with ipvlan networking

You are using LXD containers and you want a container (or more) to use an IP address from the LAN (or, get an IP address just like the host does).

LXD currently supports four ways to do that, and depending on your needs, you select the appropriate way.

1. Using macvlan. See https://blog.simos.info/how-to-make-your-lxd-container-get-ip-addresses-from-your-lan/
2. Using bridged. See https://blog.simos.info/how-to-make-your-lxd-containers-get-ip-addresses-from-your-lan-using-a-bridge/
3. Using routed. See https://blog.simos.info/how-to-get-lxd-containers-get-ip-from-the-lan-with-routed-network/
4. Using ipvlan. It is this tutorial, you are reading it now.

Why use the ipvlan networking?

You would use the ipvlan networking if you want to expose containers to the local network (LAN, or the Internet if you are using an Internet server, and have allocated several public IPs).

Any containers with ipvlan will appear on the network to have the MAC address of the host. Therefore, this will work even when you use it on your laptop that is connected to the network over WiFi (or any router with port security). That is, you can use ipvlan when macvlan and bridged cannot work.

You have to use static network configuration for these containers. Which means,

1. You need to make sure that the IP address on the network that you give to the ipvlan container, will not be assigned by the router in the future. Otherwise, there will be an IP conflict. You can do so if you go into the configuration of the router, and specify that the IP address is in use.
2. The container (i.e. the services running in the container) should not be performing changes to the network interface.

If you use some special Linux distribution, you can verify whether your LXD installation supports ipvlan by running the following command:

$ lxc info
...
api_extensions:
...
- container_nic_ipvlan
- container_nic_ipvlan_gateway
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
...
  lxc_features:
    network_ipvlan: "true"
...
$

Special requirements for container images

The default network configuration in Ubuntu 18.04 or newer is to use netplan and get eth0 to use DHCP for the configuration. The way netplan does this, messes up with ipvlan, so we are using a workaround. Depending on the Linux distribution in the container, you may need special configuration. The Ubuntu workaround is based on cloud-init, so it is the whole section for cloud-init in the profile below. Below is the list of LXD profiles per Linux distribution in the container image.

1. Ubuntu container images
2. CentOS container images
3. Debian container images

ipvlan LXD profile for Ubuntu container images

Here is the ipvlan profile, which has been tested on Ubuntu. Create a profile with this name. Then, for each container that uses the ipvlan network, we will be creating a new individual profile based on this initial profile. The reason why we create such individual profiles, is that we need to hard-code the IP address in them. Below, in bold, you can see the values that changes, specifically, the IP address (in two locations, replace with your own public IP addresses), the parent interface (on the host), and the nameserver IP address (that one is a public DNS server from Google). You can create an empty profile, then edit it and replace the existing content with the following (lxc profile create ipvlan, lxc profile edit ipvlan).

config:
  user.network-config: |
    #cloud-config
    version: 2
    ethernets:
      eth0:
        addresses:
          - 192.168.1.200/32
        dhcp4: no
        dhcp6: no
        nameservers:
          addresses: [8.8.8.8, 1.1.1.1]
        routes:
         - to: 0.0.0.0/0
           via: 169.254.0.1
           on-link: true
description: "ipvlan LXD profile"
devices:
  eth0:
    ipv4.address: 192.168.1.200
    nictype: ipvlan
    parent: enp3s0
    type: nic
name: ipvlan
used_by:

We are going to make copies of the ipvlan profile to individual new ones, one for each IP address. Therefore, let’s create the LXD profiles for 192.168.1.200 and 192.168.1.201. When you edit them

$ lxc profile copy ipvlan ipvlan_192.168.1.200
$ EDITOR=nano lxc profile edit ipvlan_192.168.1.200
$ lxc profile copy ipvlan ipvlan_192.168.1.201
$ EDITOR=nano lxc profile edit ipvlan_192.168.1.201

Skip to the next main section to test the profile.

ipvlan LXD profile for Debian container images

The following is an alternative LXD ipvaln profile that can be used on Debian 10 (buster). It might be useful for other Linux distributions as well. If this specific LXD profile works for a distribution other than Debian, please report it below so that I can update the post. It explicitly makes the container not to set network configuration through DHCP. It further uses cloud-init instructions to manually create a /etc/resolv.conf because without DHCP there wouldn’t be such a file in the container. The suggested DNS server is 8.8.8.8 (Google), and you may change if you would like. In bold, you can see the two items that you need to update for your case; the IP address for the container, and the network interface of the host that this container will attach to (through ipvlan). Note that without the dhcp4: false instruction in the following, the container will take a minute or two until it completes the startup. That is, the container tries to get a DHCP lease until it times out, and then cloud-init will eventually setup the nameserver.

config:
  user.network-config: |
    #cloud-config
    version: 2
    ethernets:
        eth0:
          dhcp4: false
          dhcp6: false
  user.user-data: |
    #cloud-config
    bootcmd:
      - echo 'nameserver 8.8.8.8' > /etc/resolvconf/resolv.conf.d/tail
      - systemctl restart resolvconf
description: ipvlan profile for Debian container images
devices:
  eth0:
    ipv4.address: 192.168.1.201
    name: eth0
    nictype: ipvlan
    parent: enp3s0
    type: nic
name: ipvlan_debian

You can launch such a Debian container with ipvlan using a command line like the following.

lxc launch images:debian/10/cloud mydebian --profile default --profile ipvlan_debian

Note that for Debian 11 (currently not released yet) the above does not work. If you can figure out a way to make it work for Debian 11, please write a comment.

ipvlan LXD profile for Fedora container images

The following is an alternative LXD routed profile that can be used on Fedora. It might be useful for other Linux distributions as well. If this specific LXD profile works for a distribution other than Fedora, please report it below so that I can update the post. The profile has two sections; the cloud-init section that configures once the networking in the container using NetworkManager, and the LXD network configuration that directs LXD on how to setup the routed networking on the host. The suggested DNS server is 8.8.8.8 (Google), and you may change if you would like with other free public DNS servers. In bold, you can see the two items that you need to update for your case; the IP address for the container, and the network interface of the host that this container will attach to (through ipvlan).

Note that you would launch the container with a command line

lxc launch images:fedora/33/cloud myfedora --profile default --profile ipvlan_fedora

config:
  user.user-data: |
    #cloud-config
    bootcmd:
      - nmcli connection modify "System eth0" ipv4.addresses 192.168.1.202/32
      - nmcli connection modify "System eth0" ipv4.gateway 169.254.0.1
      - nmcli connection modify "System eth0" ipv4.dns 8.8.8.8
      - nmcli connection modify "System eth0" ipv4.method manual
      - nmcli connection down "System eth0"
      - nmcli connection up "System eth0"
description: Default LXD profile
devices:
  eth0:
    ipv4.address: 192.168.1.202
    name: eth0
    nictype: routed
    parent: enp3s0
    type: nic
name: ipvlan_fedora

Using the ipvlan networking in LXD

We create a container called myipvlan using the default profile and on top of that the ipvlan profile.

$ lxc launch ubuntu:20.04 myipvlan --profile default --profile ipvlan
Creating myipvlan
Starting myipvlan
$ lxc list myipvlan 
+----------+---------+----------------------+-----------+-----------+
|   NAME   |  STATE  |         IPV4         |   TYPE    | SNAPSHOTS |
+----------+---------+----------------------+-----------+-----------+
| myipvlan | RUNNING | 192.168.1.200 (eth0) | CONTAINER | 0         |
+----------+---------+----------------------+-----------+-----------+
$ 

According to LXD, the container has configured its IP address that was packaged into the cloud-init configuration.

Get a shell into the container and ping

1. other IP addresses on your LAN
2. an Internet host such as www.google.com.

Here is a test try using a Fedora container image.

$ lxc launch images:fedora/33/cloud myfedora --profile default --profile ipvlan_fedora
Creating myfedora
Starting myfedora                         
$ lxc list myfedora
+----------+---------+----------------------+-----------+-----------+
|   NAME   |  STATE  |         IPV4         |   TYPE    | SNAPSHOTS |
+----------+---------+----------------------+-----------+-----------+
| myfedora | RUNNING | 192.168.1.202 (eth0) | CONTAINER | 0         |
+----------+---------+----------------------+-----------+-----------+
$ lxc shell myfedora
[root@myfedora ~]# ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=12.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=12.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=111 time=12.1 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 12.148/110.215/201.306/117.007 ms
[root@myfedora ~]# logout
$ 

Conclusion

We have seen how to setup and use ipvlan in LXD, when launching Ubuntu and Fedora container images (Debian is still pending, if you figure it out, please write a comment).

We show how to use LXD profiles to setup easily the creation of the container, and in the profile we add the IP address of the container. This means that for each container we would need to create individual LXD profiles. Note that a LXD profile is attached to a container, so if you want to change it for another container, the change will apply to any existing container as well (i.e. mess). You also could create the containers without needing an additional LXD profile, by perform lxc config commands on the host, and networking commands inside the container. We do not show that here.

You get a similar result when using ipvlan and routed. I do not go into detail about the practical differences between the two.

Simos Xenitellis