Benchmarking LXD on an AMD EPYC server at packet.net

This is the fourth post in the series.

1. A closer look at AMD EPYC baremetal servers at packet.net
2. Booting up the AMD EPYC baremetal server at packet.net
   
3. Configuring LXD on the AMD EPYC baremetal server at packet.net
   
4. Benchmarking LXD on an AMD EPYC server at packet.net (this post)

There is a tool for benchmarking LXD, called lxd-benchmark. We are installing it and running it. It is part of the package lxd-tools.

ubuntu@myserver:~$ sudo apt install -y lxd-tools

Let’s see what we can benchmark with this tool.

ubuntu@myserver:~$ lxd-benchmark 
Description:
   Benchmark performance of LXD

   This tool lets you benchmark various actions on a local LXD daemon.

   It can be used just to check how fast a given LXD host is, to
   compare performance on different servers or for performance tracking
   when doing changes to the LXD codebase.

   A CSV report can be produced to be consumed by graphing software.

Usage:
   lxd-benchmark [command]

Examples:
   # Spawn 20 Ubuntu containers in batches of 4
   lxd-benchmark launch --count 20 --parallel 4

   # Create 50 Alpine containers in batches of 10
   lxd-benchmark init --count 50 --parallel 10 images:alpine/edge

   # Delete all test containers using dynamic batch size
   lxd-benchmark delete

Available Commands:
  delete      Delete containers
  help        Help about any command
  init        Create containers
  launch      Create and start containers
  start       Start containers
  stop        Stop containers

Flags:
   -h, --help Print help
   -P, --parallel Number of threads to use (default -1)
       --report-file Path to the CSV report file
       --report-label Label for the new entry in the report [default=ACTION]
       --version Print version number

Use "lxd-benchmark [command] --help" for more information about a command.
ubuntu@myserver:~$

We can mass-init (i.e. create), mass-start, mass-launch (i.e. init and then start) and mass-delete containers.

Basic examples

Launching 20 containers in batches of 4

The first example in the command line help, is to launch (init and start) 20 containers in batches of 4. The default image is the current LTS version of Ubuntu (18.04.1). Here it is.

ubuntu@myserver:~$ lxd-benchmark launch --count 20 --parallel 4
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu13
  Container backend: lxc
  Container version: 3.0.2
Test variables:
  Container count: 20
  Container mode: unprivileged
  Startup mode: normal startup
  Image: ubuntu:
  Batches: 5
  Batch size: 4
  Remainder: 0

[Oct  3 19:40:22.687] Found image in local store: c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a
[Oct  3 19:40:22.687] Batch processing start
[Oct  3 19:40:28.258] Processed 4 containers in 5.572s (0.718/s)
[Oct  3 19:40:34.459] Processed 8 containers in 11.772s (0.680/s)
[Oct  3 19:40:46.106] Processed 16 containers in 23.419s (0.683/s)
[Oct  3 19:40:53.007] Batch processing completed in 30.320s

ubuntu@myserver:~$ 

The launching speed is about 0.7 containers per second at a total of 30 seconds.

Deleting 20 containers

Let’s delete those 20 containers.

ubuntu@myserver:~$ lxd-benchmark delete
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu13
  Container backend: lxc
  Container version: 3.0.2
[Oct  3 19:45:28.908] Deleting 20 containers
[Oct  3 19:45:28.908] Batch processing start
[Oct  3 19:45:45.192] Batch processing completed in 16.284s
ubuntu@myserver:~$ 

It took about 16 seconds to delete 20 containers.

Creating 50 Alpine containers in batches of 10

Next, we create (in fact, we init) 50 Alpine containers in batches of 10. init means that we create the container but not start it yet.

ubuntu@myserver:~$ lxd-benchmark init --count 50 --parallel 10 images:alpine/edge
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu13
  Container backend: lxc
  Container version: 3.0.2

Test variables:
  Container count: 50
  Container mode: unprivileged
  Startup mode: normal startup
  Image: images:alpine/edge
  Batches: 5
  Batch size: 10
  Remainder: 0

[Oct  3 19:46:51.279] Found image in local store: c9912a815b2d0ccf91694b3c270938f280bb7cb96bdb15c95b14ca14eb268bb5
[Oct  3 19:46:51.279] Batch processing start
[Oct  3 19:46:52.098] Processed 10 containers in 0.819s (12.214/s)
[Oct  3 19:46:53.449] Processed 20 containers in 2.169s (9.219/s)
[Oct  3 19:46:55.936] Processed 40 containers in 4.657s (8.589/s)
[Oct  3 19:46:56.978] Batch processing completed in 5.699s
ubuntu@myserver:~$ 

It took about 6 seconds to init 50 Alpine containers at a rate of about 9 containers per second.

Finally, we delete all containers as shown earlier. It took 15.6s to delete those 50 containers.

Launching 100 Ubuntu containers

The next benchmark is to launch one hundred Ubuntu containers. Those containers will get IP addresses from the lxdbr0 bridge.

Here is the command to launch 100 Ubuntu 18.04 containers. We select to launch in batches of 24 containers.

ubuntu@myserver:~$ lxd-benchmark launch --count 100 --parallel 24 ubuntu:18.04
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu13
  Container backend: lxc
  Container version: 3.0.2

Test variables:
   Container count: 100
   Container mode: unprivileged
   Startup mode: normal startup
   Image: ubuntu:18.04
   Batches: 4
   Batch size: 24
   Remainder: 4

[Oct  4 18:04:18.836] Found image in local store: d72ae2e5073f20450c5260e6f227484c23452a46c6bb553ffe6be55e48602bb4
[Oct  4 18:04:18.836] Batch processing start
[Oct  4 18:04:32.907] Processed 24 containers in 14.070s (1.706/s)
[Oct  4 18:04:51.117] Processed 48 containers in 32.281s (1.487/s)
[Oct  4 18:05:31.630] Processed 96 containers in 72.793s (1.319/s)
[Oct  4 18:05:40.542] Batch processing completed in 81.706s
ubuntu@myserver:~$

The speed was about 1.3 launched containers per second, created in about a minute and 20 seconds.

Then, we stop these container and finally delete them.

ubuntu@myserver:~$ lxd-benchmark stop
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-36-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu16.3
  Container backend: lxc
  Container version: 3.0.2

[Oct  4 18:11:31.106] Stopping 100 containers
[Oct  4 18:11:31.106] Batch processing start
[Oct  4 18:11:38.386] Processed 48 containers in 7.280s (6.593/s)
[Oct  4 18:11:44.291] Processed 96 containers in 13.185s (7.281/s)
[Oct  4 18:11:47.516] Batch processing completed in 16.410s

ubuntu@myserver:~$ lxd-benchmark delete
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-36-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu16.3
  Container backend: lxc
  Container version: 3.0.2

[Oct  4 18:12:22.896] Deleting 100 containers
[Oct  4 18:12:22.896] Batch processing start
[Oct  4 18:12:47.550] Processed 48 containers in 24.653s (1.947/s)
[Oct  4 18:13:15.700] Processed 96 containers in 52.804s (1.818/s)
[Oct  4 18:13:19.553] Batch processing completed in 56.657

The 100 container were stopped in 16 seconds and then deleted in 57 seconds.

Launching 1200 Alpine containers

The next benchmark is to launch many Alpine containers. These containers will not get IP addresses from the lxdbr0 bridge because there is a hard Linux kernel limit to the number of IP addresses on a single bridge, and that limit is 1024. It is in BR_MAX_PORTS. One option would be to recompile the kernel and increase the value. Another option would be to use something other than a bridge (like openvswitch).

To disable networking in LXD, we are going to remove the networking configuration from the LXD default profile. Here is the command to remove the configuration. At the end of the benchmark, we show how to add again.

ubuntu@myserver:~$ lxc profile device remove default eth0
Device eth0 removed from default
ubuntu@myserver:~$ 

Here is the command to launch 1200 Alpine containers.

ubuntu@myserver:~$ lxd-benchmark launch --count 1200 --parallel 24 images:alpine/edge
Test environment:
  Server backend: lxd
  Server version: 3.0.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu13
  Container backend: lxc
  Container version: 3.0.2

Test variables:
   Container count: 1200
   Container mode: unprivileged
   Startup mode: normal startup
   Image: ubuntu:18.04
   Batches: 50
   Batch size: 24
   Remainder: 0

[Oct  4 19:12:24.761] Found image in local store: 87530b2678f03cbc51a7e1732acfed679c4a15ba1939197cbda733346ef28ad9
[Oct  4 19:12:24.761] Batch processing start
[Oct  4 19:12:27.209] Processed 24 containers in 2.447s (9.806/s)
[Oct  4 19:12:29.892] Processed 48 containers in 5.130s (9.356/s)
[Oct  4 19:12:35.286] Processed 96 containers in 10.525s (9.121/s)
[Oct  4 19:12:47.344] Processed 192 containers in 22.583s (8.502/s)
[Oct  4 19:13:11.431] Processed 384 containers in 46.669s (8.228/s)
[Oct  4 19:14:03.468] Processed 768 containers in 98.707s (7.781/s)
[Oct  4 19:15:13.451] Batch processing completed in 168.690s
ubuntu@myserver:~$ 

The speed was about 8 containers per second, created in less than 3 minutes.

Now, let’s delete them and bring back the LXD installation to the original state.

ubuntu@myserver:~$ lxd-benchmark delete
Test environment:
Server backend: lxd
Server version: 3.0.2
Kernel: Linux
Kernel architecture: x86_64
Kernel version: 4.15.0-36-generic
Storage backend: zfs
Storage version: 0.7.5-1ubuntu16.3
Container backend: lxc
Container version: 3.0.2

[Oct 4 19:16:56.644] Deleting 1200 containers
[Oct 4 19:16:56.644] Batch processing start
[Oct 4 19:17:00.827] Processed 48 containers in 4.182s (11.477/s)
[Oct 4 19:17:05.152] Processed 96 containers in 8.508s (11.284/s)
[Oct 4 19:17:13.448] Processed 192 containers in 16.804s (11.426/s)
[Oct 4 19:17:28.437] Processed 384 containers in 31.793s (12.078/s)
[Oct 4 19:17:53.225] Processed 768 containers in 56.581s (13.574/s)
[Oct 4 19:18:13.147] Batch processing completed in 76.503s
ubuntu@myserver:~

We managed to delete all the container in a minute and 15 seconds. The deletion speed was about 14 containers per second.

Here we are adding back the eth0 device in the LXD default profile.

ubuntu@myserver:~$ lxc profile device add default eth0 nic nictype=bridged parent=lxdbr0
Device eth0 added to default
ubuntu@myserver:~$ 

Summary

We installed lxd-benchmark, a tool that helps perform benchmarks for LXD. We run the standard benchmarks that are listed in the command’s help. Then, we launched 100 LXD containers and cleared them up. Finally, we launched 1200 LXD container (without networking due to external caveats) and then deleted them.

List of other posts in this series

1. A closer look at AMD EPYC baremetal servers at packet.net
2. Booting up the AMD EPYC baremetal server at packet.net
   
3. Configuring LXD on the AMD EPYC baremetal server at packet.net
   
4. Benchmarking LXD on an AMD EPYC server at packet.net (this post)

Simos Xenitellis