Netdata does real-time health monitoring and performance troubleshooting for systems and applications. It helps you instantly diagnose slowdowns and anomalies in your infrastructure with thousands of metrics, interactive visualizations, and insightful health alarms.
When you set it up on your system, Netdata sets up a Web page where you can view real-time information, including CPU load, network traffic and lots more. It looks like the following.
Looks good, let’s install it! Normally, you would install Netdata on the host so that you can have visibility of the whole host. In addition, it makes sense to install on the host because since 2016, Netdata actually understands LXC/LXD containers.
However, in this post we install Netdata in a LXD container. An unprivileged LXD container. The purpose of this exercise is to get to know Netdata before installing on the host. Because your host is too important to install any software before testing them first in a LXD container.
In addition to testing the software before installing on the host, we get to see in practice how well is a container isolated from the host. That is, can the container reveal to us any significant information about the host? Obviously, it is possible to deduce whether the host is under lots of load, but let’s see this play in front of us.
Installing Netdata in a container
We launch a container, get a shell and install Netdata. We choose the installation command that compiles Netdata for us. The other download option to use static amd64 packages will be downloading pre-compiled packages for us. The installation is uneventful; we need to press Enter a few times until the installation gets completed.
$ lxc launch ubuntu:18.04 netdata
Creating netdata
Starting netdata
$ lxc exec netdata -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
ubuntu@netdata:~$ bash <(curl -Ss https://my-netdata.io/kickstart.sh)
...
We detected these:
Distribution : ubuntu
Version : 18.04
Codename : 18.04.3 LTS (Bionic Beaver)
Package Manager : install_apt_get
Packages Tree : debian
Detection Method: /etc/os-release
Default Python v: 2
...
IMPORTANT <<
Please make sure your system is up to date
by running: apt-get update
apt-get install autoconf
apt-get install autoconf-archive
apt-get install automake
apt-get install gcc
apt-get install libjudy-dev
apt-get install liblz4-dev
apt-get install libmnl-dev
apt-get install libssl-dev
apt-get install libuv1-dev
apt-get install make
apt-get install pkg-config
apt-get install python
apt-get install uuid-dev
apt-get install zlib1g-dev
Press ENTER to run it >
Netdata explains that it is going to install a set of packages. It does not do a -y in the apt command, which means we need to press Enter during each apt install. Ideally, all installations could be merged together to a single command. Below is the second time you need to press Enter. It is good to take note of the locations of the files. When you press Enter, Netdata will start the compilation and then the installation.
... ^ |.-. .-. .-. .-. . netdata | '-' '-' '-' '-' real-time performance monitoring, done right! +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---> You are about to build and install netdata to your system. It will be installed at these locations: the daemon at /usr/sbin/netdata config files in /etc/netdata web files in /usr/share/netdata plugins in /usr/libexec/netdata cache files in /var/cache/netdata db files in /var/lib/netdata log files in /var/log/netdata pid file at /var/run/netdata.pid logrotate file at /etc/logrotate.d/netdata This installer allows you to change the installation path. Press Control-C and run the same command with --help for help. Press ENTER to build and install netdata to your system >
Here is Netdata having completed the installation. It listens on port 19999 on all interface, so we can access from the host using the IP address of the container. It is already running, and we can see the commands to stop and start. There is a script to uninstall and a script to update Netdata. There is a cron service that automatically checks for updates and if there is, it will email us. Which means that since there is no MTA in the container, we will not get any emails.
netdata by default listens on all IPs on port 19999, so you can access it with: http://this.machine.ip:19999/ To stop netdata run: systemctl stop netdata To start netdata run: systemctl start netdata Uninstall script copied to: /usr/libexec/netdata/netdata-uninstaller.sh --- Install (but not enable) netdata updater tool --- Update script is located at /usr/libexec/netdata/netdata-updater.sh --- Check if we must enable/disable the netdata updater tool --- Adding to cron Auto-updating has been enabled. Updater script linked to: /etc/cron.daily/netdata-update netdata-updater.sh works from cron. It will trigger an email from cron only if it fails (it should not print anything when it can update netdata). --- Wrap up environment set up --- Preparing .environment file Setting netdata.tarball.checksum to 'new_installation' --- We are done! --- ^ |.-. .-. .-. .-. .-. . netdata .-. .- | '-' '-' '-' '-' '-' is installed and running now! -' '-' +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---> enjoy real-time performance and health monitoring… OK ubuntu@netdata:~$
We are ready to start using Netdata.
Using Netdata
The IP address of the Netdata container in my case is 10.10.10.199. Therefore, I use the browser on my host to access the URL http://10.10.10.199:19999/
Here is how the first screen looks like.
Before doing more testing, click on Settings and change to the following. We change the When to refresh the charts to Always. By doing so, the charts are refreshed even if the browser does not have the focus. This is handy when we are testing something in a terminal window and we want immediate feedback from Netdata.
We can now view all the charts and try to affect them by, for example, creating load or traffic in the container and on the host. In this way, we can sense what information may get leaked from the host to the container, if any.
Interpreting the Netdata charts
Let’s see again the screenshot of Netdata, of the first screen.
First, we can see that the actual use of the swap partition is shown in the container. Should this be shown, should it not? Does it make a practical difference? This could be hidden through lxcfs.
Second, there is significant inbound and outbound network traffic. The container is idle. Could that be traffic from the host appearing (as traffic count) in the container? Let’s tcpdump it. This is actually the Netdata traffic between the host and the container. The container cannot see traffic that belongs to the host (or other containers), nor packet counts from others.
Third, the used RAM is the percentage of the container’s used RAM over the total available RAM to the container. That is, if we did not set a total memory limit to the container, the total RAM is the RAM of the host. LXD does not reveal how much RAM is used on the host or other containers. If you encounter software that checks how much RAM is reported as available and uses as much as possible, then this software will have a bad time. I think the mysql package was doing this a few years back, and ended up not being able to start.
Affecting the charts
We can create CPU load, we can cause disk reads and disk writes and so on. Here is a way to cause CPU load and disk read. This command will read files and use the CPU to compute the SHA-1 sum.
ubuntu@netdata:~$ sha1sum /usr/bin/*
sha1sum /usr/bin/*.If you run the command again, and your system has enough RAM, you will probably not notice any more disk reads, just CPU load. Because the files get cached in the disk cache. To test the allocation of RAM, we could write a program to malloc() big amounts of memory.
But why do all these while we can go pro. Let’s install stress-ng, both on the host and the netdata container.
ubuntu@netdata:~$ sudo apt install stress-ng
Stress memory and CPU
Run the following to allocate 1GB of RAM. By default, stress-ng uses workers to stress the resource you are allocating.
ubuntu@netdata:~$ stress-ng --timeout 3s --vm 1G stress-ng: info: [4388] dispatching hogs: 1 vm stress-ng: info: [4388] successful run completed in 3.01s ubuntu@netdata:~$
Try running this in the container and on the host.
Stress the I/O
Run the following. 10 is the number of workers. This affects the disk writes.
ubuntu@netdata:~$ stress-ng --timeout 3s --io 10 stress-ng: info: [4467] dispatching hogs: 10 io stress-ng: info: [4467] successful run completed in 4.54s ubuntu@netdata:~$
Conclusion
We have setup netdata in a LXD container in order to view real-time information but most importantly to get to know how the software works, before we install on the host. Do more tests and get a good feel as to what is the view of your system from the point of view of a LXD container.
It is very useful to try software in a system container before installing on the host. In this specific case, we cannot make effective use of netdata when it is installed in a container because the tool needs to access too many metrics on the host. These metrics are over 2000 in number, and it does not look practical to expose them to the container.
The next step to testing Netdata in a LXD system container, is to get Netdata to graduate and then install on the host. That’s a subject of a future post. It is am important subject because Netdata understands LXD containers, and is able to provide real-time information for each container.
Recent Comments