How to setup Netdata in a LXD container for real-time monitoring

Netdata does real-time health monitoring and performance troubleshooting for systems and applications. It helps you instantly diagnose slowdowns and anomalies in your infrastructure with thousands of metrics, interactive visualizations, and insightful health alarms.

When you set it up on your system, Netdata sets up a Web page where you can view real-time information, including CPU load, network traffic and lots more. It looks like the following.

Netdata running on a system. On top it shows the dashboard. On the right, the system elements that are monitored.

Looks good, let’s install it! Normally, you would install Netdata on the host so that you can have visibility of the whole host. In addition, it makes sense to install on the host because since 2016, Netdata actually understands LXC/LXD containers.

However, in this post we install Netdata in a LXD container. An unprivileged LXD container. The purpose of this exercise is to get to know Netdata before installing on the host. Because your host is too important to install any software before testing them first in a LXD container.

In addition to testing the software before installing on the host, we get to see in practice how well is a container isolated from the host. That is, can the container reveal to us any significant information about the host? Obviously, it is possible to deduce whether the host is under lots of load, but let’s see this play in front of us.

Installing Netdata in a container

We launch a container, get a shell and install Netdata. We choose the installation command that compiles Netdata for us. The other download option to use static amd64 packages will be downloading pre-compiled packages for us. The installation is uneventful; we need to press Enter a few times until the installation gets completed.

$ lxc launch ubuntu:18.04 netdata
 Creating netdata
 Starting netdata
$ lxc exec netdata -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo ".
 See "man sudo_root" for details.

ubuntu@netdata:~$ bash <(curl -Ss
We detected these:
 Distribution    : ubuntu
 Version         : 18.04
 Codename        : 18.04.3 LTS (Bionic Beaver)
 Package Manager : install_apt_get
 Packages Tree   : debian
 Detection Method: /etc/os-release
 Default Python v: 2 
        IMPORTANT << 
         Please make sure your system is up to date
         by running:   apt-get update      
 apt-get install autoconf 
 apt-get install autoconf-archive 
 apt-get install automake 
 apt-get install gcc 
 apt-get install libjudy-dev 
 apt-get install liblz4-dev 
 apt-get install libmnl-dev 
 apt-get install libssl-dev 
 apt-get install libuv1-dev 
 apt-get install make 
 apt-get install pkg-config 
 apt-get install python 
 apt-get install uuid-dev 
 apt-get install zlib1g-dev 
 Press ENTER to run it > 

Netdata explains that it is going to install a set of packages. It does not do a -y in the apt command, which means we need to press Enter during each apt install. Ideally, all installations could be merged together to a single command. Below is the second time you need to press Enter. It is good to take note of the locations of the files. When you press Enter, Netdata will start the compilation and then the installation.

|.-.   .-.   .-.   .-.   .  netdata                                        
|   '-'   '-'   '-'   '-'   real-time performance monitoring, done right!  

You are about to build and install netdata to your system.

It will be installed at these locations:

the daemon     at /usr/sbin/netdata
config files   in /etc/netdata
web files      in /usr/share/netdata
plugins        in /usr/libexec/netdata
cache files    in /var/cache/netdata
db files       in /var/lib/netdata
log files      in /var/log/netdata
pid file       at /var/run/
logrotate file at /etc/logrotate.d/netdata 

This installer allows you to change the installation path.
Press Control-C and run the same command with --help for help.

Press ENTER to build and install netdata to your system > 

Here is Netdata having completed the installation. It listens on port 19999 on all interface, so we can access from the host using the IP address of the container. It is already running, and we can see the commands to stop and start. There is a script to uninstall and a script to update Netdata. There is a cron service that automatically checks for updates and if there is, it will email us. Which means that since there is no MTA in the container, we will not get any emails.

netdata by default listens on all IPs on port 19999,
so you can access it with:


To stop netdata run:

  systemctl stop netdata

To start netdata run:

  systemctl start netdata

Uninstall script copied to: /usr/libexec/netdata/

--- Install (but not enable) netdata updater tool --- 
 Update script is located at /usr/libexec/netdata/

--- Check if we must enable/disable the netdata updater tool --- 
Adding to cron
Auto-updating has been enabled. Updater script linked to: /etc/cron.daily/netdata-update works from cron. It will trigger an email from cron
only if it fails (it should not print anything when it can update netdata).

--- Wrap up environment set up --- 
Preparing .environment file
Setting netdata.tarball.checksum to 'new_installation'

--- We are done! --- 

   |.-.   .-.   .-.   .-.   .-.   .  netdata                          .-.   .-
   |   '-'   '-'   '-'   '-'   '-'   is installed and running now!  -'   '-'  

enjoy real-time performance and health monitoring…



We are ready to start using Netdata.

Using Netdata

The IP address of the Netdata container in my case is Therefore, I use the browser on my host to access the URL

Here is how the first screen looks like.

View of Netdata running in a LXD container. In my case, the URL was

Before doing more testing, click on Settings and change to the following. We change the When to refresh the charts to Always. By doing so, the charts are refreshed even if the browser does not have the focus. This is handy when we are testing something in a terminal window and we want immediate feedback from Netdata.

The Settings dialog in Netdata. We see the Performance tab and changed to refresh the charts to Always.

We can now view all the charts and try to affect them by, for example, creating load or traffic in the container and on the host. In this way, we can sense what information may get leaked from the host to the container, if any.

Interpreting the Netdata charts

Let’s see again the screenshot of Netdata, of the first screen.

Netdata charts, in a LXD system container.

First, we can see that the actual use of the swap partition is shown in the container. Should this be shown, should it not? Does it make a practical difference? This could be hidden through lxcfs.

Second, there is significant inbound and outbound network traffic. The container is idle. Could that be traffic from the host appearing (as traffic count) in the container? Let’s tcpdump it. This is actually the Netdata traffic between the host and the container. The container cannot see traffic that belongs to the host (or other containers), nor packet counts from others.

Third, the used RAM is the percentage of the container’s used RAM over the total available RAM to the container. That is, if we did not set a total memory limit to the container, the total RAM is the RAM of the host. LXD does not reveal how much RAM is used on the host or other containers. If you encounter software that checks how much RAM is reported as available and uses as much as possible, then this software will have a bad time. I think the mysql package was doing this a few years back, and ended up not being able to start.

Affecting the charts

We can create CPU load, we can cause disk reads and disk writes and so on. Here is a way to cause CPU load and disk read. This command will read files and use the CPU to compute the SHA-1 sum.

ubuntu@netdata:~$ sha1sum /usr/bin/*
Disk reads and CPU load when running sha1sum /usr/bin/*.

If you run the command again, and your system has enough RAM, you will probably not notice any more disk reads, just CPU load. Because the files get cached in the disk cache. To test the allocation of RAM, we could write a program to malloc() big amounts of memory.

But why do all these while we can go pro. Let’s install stress-ng, both on the host and the netdata container.

ubuntu@netdata:~$ sudo apt install stress-ng

Stress memory and CPU

Run the following to allocate 1GB of RAM. By default, stress-ng uses workers to stress the resource you are allocating.

ubuntu@netdata:~$ stress-ng --timeout 3s --vm 1G
stress-ng: info:  [4388] dispatching hogs: 1 vm
stress-ng: info:  [4388] successful run completed in 3.01s

Try running this in the container and on the host.

Stress the I/O

Run the following. 10 is the number of workers. This affects the disk writes.

ubuntu@netdata:~$ stress-ng --timeout 3s --io 10
stress-ng: info:  [4467] dispatching hogs: 10 io
stress-ng: info:  [4467] successful run completed in 4.54s


We have setup netdata in a LXD container in order to view real-time information but most importantly to get to know how the software works, before we install on the host. Do more tests and get a good feel as to what is the view of your system from the point of view of a LXD container.

It is very useful to try software in a system container before installing on the host. In this specific case, we cannot make effective use of netdata when it is installed in a container because the tool needs to access too many metrics on the host. These metrics are over 2000 in number, and it does not look practical to expose them to the container.

The next step to testing Netdata in a LXD system container, is to get Netdata to graduate and then install on the host. That’s a subject of a future post. It is am important subject because Netdata understands LXD containers, and is able to provide real-time information for each container.

Permanent link to this article:

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: