How to run Julia on Jupyter in a LXD container

Julia is a computer language well-suited for scientists, engineers and students. I wrote the following introduction a couple of days ago.

In this post we are going to see how to use Julia in the Jupyter interactive environment. This means that you run Julia commands in your browser and get the output in your browser. It is much more comfortable than using a terminal window and the command line.

First, we create a LXD container and install Julia in there. Then, we add the IJulia package which is the package for Jupyter support. Subsequently, we expose the network port of Jupyter to the host so that we can view it with our Web browser.

Installing Julia in a LXD container

First, launch a ubuntu:18.04 container. We call the container julia.

$ lxc launch ubuntu:18.04 julia
Creating julia
Starting julia

Then, get a shell into the container and install Julia according to the following abbreviated instructions.

$ lxc exec julia -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@julia:~$

Here we are installing Julia.

ubuntu@julia:~$ wget https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.0-linux-x86_64.tar.gz
ubuntu@julia:~$ tar xfa julia-1.0.0-linux-x86_64.tar.gz
ubuntu@julia:~$ ls -l julia-1.0.0/bin/
total 28
-rwxr-xr-x 1 ubuntu ubuntu 47280 Aug 9 01:02 julia
ubuntu@julia:~$ ./julia-1.0.0/bin/julia
| Documentation: https://docs.julialang.org
|
| Type "?" for help, "]?" for Pkg help.
|
| Version 1.0.0 (2018-08-08)
| Official https://julialang.org/ release
|

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)

julia> exit()
ubuntu@julia:~$

Now, Julia is installed in the LXD container and we are able to move to the next step, setting up Jupyter.

Setting up the Julia kernel for Jupyter

We follow the instructions at the IJulia Julia package page. Specifically,

julia> using Pkg
julia> Pkg.add("IJulia")
Cloning default registries into /home/ubuntu/.julia/registries
Cloning registry General from "https://github.com/JuliaRegistries/General.git"
Updating registry at ~/.julia/registries/General
Updating git-repo https://github.com/JuliaRegistries/General.git
Resolving package versions…
Installed SoftGlobalScope ─ v1.0.5
Installed VersionParsing ── v1.1.2
Installed Conda ─────────── v1.0.1
Installed ZMQ ───────────── v1.0.0
Installed BinaryProvider ── v0.4.2
Installed IJulia ────────── v1.11.1
Installed Compat ────────── v1.1.0
Installed MbedTLS ───────── v0.6.0
Installed JSON ──────────── v0.19.0
Updating ~/.julia/environments/v1.0/Project.toml
[7073ff75] + IJulia v1.11.1
Updating ~/.julia/environments/v1.0/Manifest.toml
... UUIDs [4ec0a83e] + Unicode Building Conda ──→ ~/.julia/packages/Conda/m7vem/deps/build.log Building ZMQ ────→ ~/.julia/packages/ZMQ/ABGOx/deps/build.log Building MbedTLS → ~/.julia/packages/MbedTLS/Qo8TN/deps/build.log Building IJulia ─→ ~/.julia/packages/IJulia/k5o7j/deps/build.log julia> using IJulia [ Info: Precompiling IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a]

The necessary software has been installed. Now we launch the Jupyter Notebook Web service in the following way.

julia> notebook(detached=true)
Process(setenv(/home/ubuntu/.julia/packages/Conda/m7vem/deps/usr/bin/jupyter notebook --NotebookApp.iopub_data_rate_limit=2147483647; dir="/home/ubuntu"), ProcessRunning)

julia> exit()
ubuntu@julia:~/julia-1.0.0/bin$

We are almost ready to use our Web browser to open the Notebook page.

Getting the Jupyter Notebook token

By default, the Jupyter Notebook requires the users to be authenticated and in this case, you need to have a special token in order to access the Notebook. Here is the token, and URL to use to get directly access to the Notebook.

ubuntu@julia:~$ ./.julia/packages/Conda/m7vem/deps/usr/bin/jupyter notebook list
Currently running servers:
http://localhost:8888/?token=40306ed2c602ecc5e64d2619c2d7f03aa8534f44106c353e :: /home/ubuntu
ubuntu@julia:~$

Note the the Jupyter Notebook is listening on the container’s localhost at port 8888. But how can our host’s browser access that port? See next.

Creating a LXD proxy device to the Julia container

We need to proxy the TCP port 8888 at the container (interface: lo) to the loopback of the host. Here is how we do this with the LXD proxy device. We create a proxy device called port8888, which links together the container’s TCP port 8888 to the host’s TCP port 8888. The exist port is the one in the container, and LXD creates the port 8888 on the host. In this way, the Web browser on your desktop can access the Jupyter Notebook with Julia, using the exact URL from the previous section.

$ lxc config device add julia port8888 proxy listen=tcp:localhost:8888 connect=tcp:localhost:8888
Device port8888 added to julia

If instead you want to make the Jupyter Notebook also accessible by any computer at your local network, you can create instead the following proxy device. We remove the old device and create a new one. With changed the listen option, and specified to LXD to listen on all network interfaces on the host.

$ lxc config device remove julia port8888
Device port8888 removed from julia
$ lxc config device add julia port8888 proxy listen=tcp:0.0.0.0:8888 connect=tcp:localhost:8888
Device port8888 added to julia

Using the Jupyter Notebook for Julia

We have done the hard work, let’s open up the browser and visit the Jupyter Notebook URL. In my case, the URL that was given earlier is

http://localhost:8888/?token=40306ed2c602ecc5e64d2619c2d7f03aa8534f44106c353e

Here is the web page.

Jupyter Notebook first page.

That was the first page. We have not created a Notebook yet and we are about to create it now. Click on the New button (at the right), then select Julia 1.0.0.

Creating a new Notebook for Julia 1.0.0.

Finally, here is Julia in a Jupyter Notebook!

Julia in a Jupyter Notebook.

At this point, we can run Julia commands in here. See how to do that in the next section.

Using the Julia Jupyter Notebook

When you type a Julia command in the green box, you need to press Shift+Enter so that it gets executed. In the following screenshot, we install the Plots package.

Installing the Plots package. Notice the [*] in the screenshot.

The [*] in the screenshot means that the command is still running and we should wait for the command to complete. When the command Pkg.add(“Plots”) is completed the * will become a number, and in this case the number 3 (that is, the third command we have run).

Let’s plot something using the default plotting back-end, whichever that is.

Creating a plot in a Jupyter Notebook.

We will stop the post here. There is a workflow on how to manage Jupyter Notebooks. This, currently untitled, Notebook has been autosaved. You may set a name. You can see the Notebook in the first tab of the browser which shows the files in Jupyter.

Conclusion

We have seen how to set up a LXD container to start a Jupyter Notebook instance in order to run Julia commands. In this way, you can use the browser to manage all commands and avoid getting back to the terminal window.

In addition, you can set up Julia to use your GPU for hardware-accelerated tasks such as machine learning. This can be performed because LXD supports GPU passthrough and applications in a LXD container can have access to the GPU.

Furthermore, you can also setup Julia to run in a LXD container on a remote beefed-up computer (multiple cores, GPU(s)).

Finally, you can also setup the container so that the Jupyter Notebook gets autostarted when you reboot your computer or restart the julia LXD container. See https://blog.simos.info/how-to-install-a-node-js-app-in-a-lxd-container/ for some hints on that. For this to work, make sure to set up a username/password so that you do not need to connect to the LXD container to get the new authentication hash on the next restart.

Permanent link to this article: https://blog.simos.info/how-to-run-julia-on-jupyter-in-a-lxd-container/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: