How to create a snap for a Python app with networking using snapcraft in Ubuntu 16.04

Update #1: 6 Feb 2017  httpstat needs curl. Originally, this HowTo would freshly compile curl from the github source. Now, this HowTo shows how to reuse the existing curl package from the Ubuntu repositories.

In this post we see how to create a snap package (or, just snap) of some software in Ubuntu 16.04. A snap is a confined package, that can only do as much as we allow it. Once we create it, we can install the same snap in different Linux distributions!

Each software has different needs. This one 1) requires access to the Internet and 2) requires an additional binary to be added to the snap. And nothing else.

Here is what we will be doing today.

  1. Get to know this cool software that we are going to snap, httpstat.
  2. Setup snapcraft that helps us create the snap.
  3. Start building the snap incrementally, trial and error.
  4. Complete the httpstat snap
  5. Register the name on the Ubuntu Store
  6. Upload and release the snap to the Ubuntu Store so anyone can install it!
  7. Install and run the new snap!

About httpstat

httpstat can be found at https://github.com/reorx/httpstat and it is a network utility that shows how fast is the access to a website.

$ apt search httpstat
Sorting... Done
Full Text Search... Done

It is not available yet as an apt package, so all the better to create a snap.

Let’s get the source of httpstat and run it manually. This will give us a good idea what dependencies it may have.

$ git clone https://github.com/reorx/httpstat
Cloning into 'httpstat'...
remote: Counting objects: 251, done.
remote: Total 251 (delta 0), reused 0 (delta 0), pack-reused 251
Receiving objects: 100% (251/251), 330.04 KiB | 336.00 KiB/s, done.
Resolving deltas: 100% (138/138), done.
Checking connectivity... done.
$ cd httpstat/
$ ls
httpstat.py  httpstat_test.sh  LICENSE  Makefile  README.md  screenshot.png  setup.py
$ python httpstat.py 
Usage: httpstat URL [CURL_OPTIONS]
       httpstat -h | --help
       httpstat --version

Arguments:
  URL     url to request, could be with or without `http(s)://` prefix

Options:
  CURL_OPTIONS  any curl supported options, except for -w -D -o -S -s,
                which are already used internally.
  -h --help     show this screen.
  --version     show version.

Environments:
  HTTPSTAT_SHOW_BODY    Set to `true` to show response body in the output,
                        note that body length is limited to 1023 bytes, will be
                        truncated if exceeds. Default is `false`.
  HTTPSTAT_SHOW_IP      By default httpstat shows remote and local IP/port address.
                        Set to `false` to disable this feature. Default is `true`.
  HTTPSTAT_SHOW_SPEED   Set to `true` to show download and upload speed.
                        Default is `false`.
  HTTPSTAT_SAVE_BODY    By default httpstat stores body in a tmp file,
                        set to `false` to disable this feature. Default is `true`
  HTTPSTAT_CURL_BIN     Indicate the curl bin path to use. Default is `curl`
                        from current shell $PATH.
  HTTPSTAT_DEBUG        Set to `true` to see debugging logs. Default is `false`

$ python httpstat.py www.google.com
Connected to 216.58.206.4:80 from 192.168.1.106:37920

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.gr/?gfe_rd=cr&ei=4bWUWJAnqML68Ae3gKqYBQ
Content-Length: 258
Date: Fri, 03 Feb 2017 14:54:57 GMT

Body stored in: /tmp/tmpRGDhKE

  DNS Lookup   TCP Connection   Server Processing   Content Transfer
[    12ms    |      51ms      |       52ms        |        1ms       ]
             |                |                   |                  |
    namelookup:12ms           |                   |                  |
                        connect:63ms              |                  |
                                      starttransfer:115ms            |
                                                                 total:116ms  

We cloned the repository, then noticed that there is a httpstat.py file in there, and we ran the file. The Python script accepts as a parameter a URL and that’s it. No need for compilation, aye!

We show the httpstat output for google.com. httpstat shows how much time it took for each individual stage of the transfer, in this case (no SSL/TLS, just the redirected page) is

  1. DNS Lookup
  2. TCP Connection
  3. Server Processing
  4. Content Transfer

A system administrator would need to reduce those times. For example, they could switch to a different server, get a faster server or optimize the server software so that it delivers the content much faster.

Set up Snapcraft

snapcraft is a utility that helps us create snaps. Let’s install snapcraft.

$ sudo apt update
...
Reading state information... Done
All packages are up to date.
$ sudo apt install snapcraft
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  snapcraft
...
Preparing to unpack .../snapcraft_2.26_all.deb ...
Unpacking snapcraft (2.26) ...
Setting up snapcraft (2.26) ...
$_

In Ubuntu 16.04, snapcraft was updated today (early Feb) and has a few differences from the previous version. Make sure you have snapcraft 2.26 or newer.

Let’s create a new directory for the development of the httpstat snap and initialize it with snapcraft so that create the necessary initial files.

$ mkdir httpstat
$ cd httpstat/
$ snapcraft init
Created snap/snapcraft.yaml.
Edit the file to your liking or run `snapcraft` to get started
$ ls -l
total 4
drwxrwxr-x 2 myusername myusername 4096 Feb   3 17:09 snap
$ ls -l snap/
total 4
-rw-rw-r-- 1 myusername myusername 676 Feb   3 17:09 snapcraft.yaml
$ _

We are in this httpstat/ directory and from here we run snapcraft in order to create the snap. snapcraft will take the instructions from snap/snapcraft.yaml and do its best to create the snap.

These are the contents of snap/snapcraft.yaml:

$ cat snap/snapcraft.yaml 
name: my-snap-name # you probably want to 'snapcraft register <name>'
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil
$ _

This snap/snapcraft.yaml configuration file is actually usable and can create an (empty) snap. Let’s create this empty snap, install it, uninstall it and then clean up to the initial pristine state.

$ snapcraft 
Preparing to pull my-part 
Pulling my-part 
Preparing to build my-part 
Building my-part 
Staging my-part 
Priming my-part 
Snapping 'my-snap-name' |                                                                 
Snapped my-snap-name_0.1_amd64.snap
$ snap install my-snap-name_0.1_amd64.snap 
error: cannot find signatures with metadata for snap "my-snap-name_0.1_amd64.snap"
$ snap install my-snap-name_0.1_amd64.snap --dangerous
error: cannot perform the following tasks:
- Mount snap "my-snap-name" (unset) (snap "my-snap-name" requires devmode or confinement override)
Exit 1
$ snap install my-snap-name_0.1_amd64.snap --dangerous --devmode
my-snap-name 0.1 installed
$ snap remove my-snap-name
my-snap-name removed
$ snapcraft clean
Cleaning up priming area
Cleaning up staging area
Cleaning up parts directory
$ ls
my-snap-name_0.1_amd64.snap  snap/
$ rm my-snap-name_0.1_amd64.snap 
rm: remove regular file 'my-snap-name_0.1_amd64.snap'? y
removed 'my-snap-name_0.1_amd64.snap'
$ _

While developing the snap, we will be going through this cycle of creating the snap, testing it and then removing it. There are ways to optimize a bit this process, learn soon we will.

In order to install the snap from a .snap file, we had to use –dangerous because the snap has not been digitally signed. We also had to use –devmode because snapcraft.yaml specifies the developer mode, which is a relaxed (in terms of permissions) development mode.

Creating the httpstat snapcraft.yaml, first part

Here is the first part of the httpstat snapcraft.yaml. The first part is about the description and we are not going to change anything from here later. The second part is the snap creation instructions and it is the interesting part for trial and error.

$ cat snap/snapcraft.yaml 
name: httpstat # you probably want to 'snapcraft register <name>'
version: '1.1.3' # just for humans, typically '1.2+git' or '1.3.2'
summary: Curl statistics made simple # 79 char long summary
description: |
    httpstat is a utility that analyses show fast is a website
    when you are trying to connect to it.
    This utility is particularly useful to Web administrators

grade: stable # must be 'stable' to release into candidate/stable channels
confinement: strict # use 'strict' once you have the right plugs and slots

First, for the a name, we use httpstat. We are going to actually register it (snap register) further down in this HowTo, just before publishing the snap. If someone else is a maintainer of a package and it is a known major software package, we might need some extra steps to contact the maintainers or make our own httpstat-unofficial snap.

Second, we select a version. Instead of using the latest development version that might not work or may happen to be broken momentarily, you can pick and choose the stable branch or the latest tag.

The tag “v1.1.3” will do!

Third and fourth, we add a summary and a description from text we got from the httpstat Github page.

Fifth, we select a grade. That would be either devel or stable. This snap is going to be stable.

Sixth, we select the confinement of the snap. The best of the best is strict, which means that no much outside access is allowed outside. It is us who need to specify (afterwards) explicitly what actually is allowed. If you were to keep it at devmode, then there is no confinement at all and all is allowed just like with deb packages.

Creating the snapcraft.yaml, second part

Let’s start off with this initial version of the second part. This guide has helped us to figure out the initial stuff.

apps: 
  httpstat:
    command: bin/httpstat

parts:
  httpstat:
    plugin: python
    source: https://github.com/reorx/httpstat.git
    source-tag: v1.1.3

First, in apps section we specify that the users will be running an executable named httpstat, which can be found as bin/httpstat. Whether the real executable is bin/httpstat or something else, is determined by the parts section.

Second, in the parts section we provide instructions as to how to process this httpstat target.  We specify that we want to use the python plugin (snapcraft plugin reference), which is a plugin that performs python setup.py build and python setup.py install. Other plugins would ./configure; make; make install and so on. We specify the git URL for the source (note that it ends with .git) and the tag (snapcraft source reference).

Let’s join together the first part (it is the final version) and this second part (needs some more love),

$ cat snap/snapcraft.yaml 
name: httpstat # you probably want to 'snapcraft register <name>'
version: '1.1.3' # just for humans, typically '1.2+git' or '1.3.2'
summary: Curl statistics made simple # 79 char long summary
description: |
    httpstat is a utility that analyses show fast is a website
    when you are trying to connect to it.
    This utility is particularly useful to Web administrators

grade: stable # must be 'stable' to release into candidate/stable channels
confinement: strict # use 'strict' once you have the right plugs and slots

apps: 
  httpstat:
    command: bin/httpstat

parts:
  httpstat:
    plugin: python
    source: https://github.com/reorx/httpstat.git
    source-tag: v1.1.3

Now we can run snapcraft, create the snap, install it and test it on a website. If we get an error, we fix it and try again.

$ snapcraft 
Preparing to pull httpstat 
...                                                       
Pulling httpstat 
...
Note: checking out '0f0e653309982178302ec1d5023bda2de047a72d'.
...
Successfully downloaded httpstat
Preparing to build httpstat 
Building httpstat 
...
Installing collected packages: httpstat
Successfully installed httpstat-1.1.3
Staging httpstat 
Priming httpstat 
Snapping 'httpstat' \                                                                     
Snapped httpstat_1.1.3_amd64.snap
$ snap install httpstat_1.1.3_amd64.snap --dangerous
httpstat 1.1.3 installed
$ httpstat www.google.com
Traceback (most recent call last):
  File "/snap/httpstat/x4/bin/httpstat", line 11, in <module>
    sys.exit(main())
  File "/snap/httpstat/x4/lib/python3.5/site-packages/httpstat.py", line 155, in main
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=cmd_env)
  File "/snap/httpstat/x4/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/snap/httpstat/x4/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'curl'
Exit 1

We are almost there. Our confined httpstat snap needs to use a curl binary. Two ways to include this binary. The easy way is to stage the binary from the existing APT package of the Ubuntu repositories. The more involved way would be to freshly compile curl from source, with the opportunity to enable compilation flags to cut down the binary to the bare minimum.

If you prefer to try the more involved way, append these lines to the snapcraft.yaml file (therefore they go in the parts section) and you are set to go. These lines add a new parts subsection for curl, instructing to use autotools (./configure; make; make install) to compile the source of curl from github.

  curl:
    plugin: autotools
    source: https://github.com/curl/curl.git

In the following we are going to reuse the existing package for curl, found in the repositories. It is the same .deb file you get when you would run sudo apt install curl.

Here are the instructions to reuse an existing APT package that is found in the repositories. We use stage-packages and is put in the same level in the httpstat parts.

    stage-packages: 
        - curl

Here is the new version of snapcraft.yaml,

$ cat snap/snapcraft.yaml 
name: httpstat # you probably want to 'snapcraft register <name>'
version: '1.1.3' # just for humans, typically '1.2+git' or '1.3.2'
summary: Curl statistics made simple # 79 char long summary
description: |
    httpstat is a utility that analyses show fast is a website
    when you are trying to connect to it.
    This utility is particularly useful to Web administrators

grade: stable # must be 'stable' to release into candidate/stable channels
confinement: strict # use 'strict' once you have the right plugs and slots

apps: 
  httpstat:
    command: bin/httpstat

parts:
  httpstat:
    plugin: python
    source: https://github.com/reorx/httpstat.git
    source-tag: v1.1.3
    stage-packages: 
        - curl

Let’s run snapcraft again and produce an updated snap.

$ snapcraft
Preparing to pull httpstat 
...
Get:42 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 curl amd64 7.47.0-1ubuntu2.2 [139 kB]
...
Pulling httpstat 
...
Successfully downloaded httpstat
Preparing to build httpstat 
Building httpstat 
...
Collecting httpstat
Installing collected packages: httpstat
Successfully installed httpstat-1.1.3
Staging httpstat 
Priming httpstat 
Snapping 'httpstat' -                                                                     
Snapped httpstat_1.1.3_amd64.snap
$ _

We are ready to install the new snap and test it out.

$ snap install httpstat_1.1.3_amd64.snap --dangerous
 httpstat 1.1.3 installed
 $ httpstat google.com
 > curl -w <output-format> -D <tempfile> -o <tempfile> -s -S google.com
 curl error: curl: (6) Couldn't resolve host 'google.com'
 Exit 6

We are almost there! The snap does not have access to the Internet because of the strict confinement. We need to allow networking access while keep blocking everything else (for example, the httpstat would not have any access to our home directory).

To allow networking access to a snap, we need to specify the networking is OK for this snap. network is an interface in snaps and it is one of the many supported interfaces for snaps. There is the notion of plugs (provider of resource) and slots (consumer of resource). For most cases, like this one here, we need a plug, for network.

Here is how it is specified (and in general can have multiple items from the interface reference page for snaps),

    plugs:
      - network

Once we add these two lines, we reach the final version of snapcraft.yaml for httpstat.

name: httpstat # you probably want to 'snapcraft register <name>'
version: '1.1.3' # just for humans, typically '1.2+git' or '1.3.2'
summary: Curl statistics made simple # 79 char long summary
description: |
    httpstat is a utility that analyses show fast is a website
    when you are trying to connect to it.
    This utility is particularly useful to Web administrators

grade: stable # must be 'stable' to release into candidate/stable channels
confinement: strict # use 'strict' once you have the right plugs and slots

apps: 
  httpstat:
    command: bin/httpstat
    plugs:
      - network

parts:
  httpstat:
    plugin: python
    source: https://github.com/reorx/httpstat.git
    source-tag: v1.1.3
    stage-packages: 
      - curl

Let’s get going, produce the snap, install it and test it!

$ snapcraft clean
Cleaning up priming area
Cleaning up staging area
Cleaning up parts directory
$ snapcraft 
Preparing to pull httpstat 
...
Get:42 http://gr.archive.ubuntu.com/ubuntu xenial-updates/main amd64 curl amd64 7.47.0-1ubuntu2.2 [139 kB]
...
Pulling httpstat 
...
Successfully downloaded httpstat
Preparing to build httpstat 
Building httpstat 
...
Collecting httpstat
Installing collected packages: httpstat
Successfully installed httpstat-1.1.3
Staging httpstat 
Priming httpstat 
Snapping 'httpstat' /                                                                     
Snapped httpstat_1.1.3_amd64.snap
$ snap install httpstat_1.1.3_amd64.snap  --dangerous
httpstat 1.1.3 installed
$ httpstat https://www.google.com

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: https://www.google.gr/?gfe_rd=cr&ei=qCWWWPn3BLOT8QfItrHIDA
Content-Length: 259
Date: Sat, 04 Feb 2017 12:04:08 GMT
Alt-Svc: quic=":443"; ma=2592000; v="35,34"

Body stored in: /tmp/tmpwotxl1u0

  DNS Lookup   TCP Connection   SSL Handshake   Server Processing   Content Transfer
[     2ms    |       6ms      |     109ms     |       48ms        |        0ms       ]
             |                |               |                   |                  |
    namelookup:2ms            |               |                   |                  |
                        connect:8ms           |                   |                  |
                                    pretransfer:117ms             |                  |
                                                      starttransfer:165ms            |
                                                                                 total:165ms  
$ _

And that’s it! The snap, with strict confinement, works.

When we try the snap further, we might find enhancements and other ways to make it better. If someone wants to help, they simply need just this small snapcraft.yaml (gist) in order to try themselves to make the snap better. The first revision of the gist is the version that compiles curl from source.

Register the snap name

Before publishing publicly a snap, you first need to create an account for the Ubuntu Store (use your Launchpad.net account or Ubuntu Single Sign On (SSO) as it is now called, if you already have one). Then run snapcraft login to log in and then, snapcraft register to register the new package name. Here we go,

$ snapcraft login
Enter your Ubuntu One SSO credentials.
Email: me@example.com
Password: **********
We strongly recommend enabling multi-factor authentication: https://help.ubuntu.com/community/SSO/FAQs/2FA
Login successful.
$ snapcraft register httpstat
Registering httpstat.
Congratulations! You're now the publisher for 'httpstat'.

That’s it. Let’s see the snap in the list of our registered snaps,

$ snapcraft list-registered
Name         Since                 Visibility    Price    Notes
httpstat     2017-02-03T17:28:45Z  public        -        -

We are now ready to publish the snap.

Publish the snap

We publish the snap by first running snapcraft push. We push the *.snap file instead of the snapcraft.yaml. The process of creating an account to pushing and releasing the snap, is described at Publishing your snap.

$ snapcraft push httpstat_1.1.3_amd64.snap 
Pushing 'httpstat_1.1.3_amd64.snap' to the store.
Uploading httpstat_1.1.3_amd64.snap [                                                              ]   0%
Uploading httpstat_1.1.3_amd64.snap [==============================================================] 100%
Ready to release!|                                                                                       
Revision 1 of 'httpstat' created.

The snap has been uploaded, but not released yet to the public in one of the four channels, stable, candidate, beta or edge. Let’s see the snap online. Log in to https://myapps.developer.ubuntu.com/

When we click on the snap name, we get the detailed page for the snap. Here it is,

The important part is where it says Package status is Ready to publish. When that happens, and when it happens immediately, it means our snap managed to pass all the automated tests and it is ready to publish. For more complicated snaps, it requires a manual review process.

There are two ways to release a snap to the public, through the Web interface of the Ubuntu Store, and using snapcraft. We examine both options.

Release the snap using the Web interface of the Ubuntu Store

As shown in the screenshot of the snap details Web page, we click where it says the revision (#1) and the version (1.1.3). It looks like «#1       1.1.3» at the screenshot above. Here is the page of the Technical details for this revision and version.

A snap is released into channels. In the screenshot above, there is Channels, and a link called Release. Click on the link for Release.

Here we select the channels to release the snap in, and click on Release. We selected all channels. The most important is stable, which means the snap is public to everyone.

If you performed the release using Ubuntu Store, skip the following subsection about releasing the snap using snapcraft.

Release the snap using snapcraft

Let’s release the snap, with snapcraft release. There are four channels, stable, candidate, beta and edge. We release the snap to all four channels.

$ snapcraft release httpstat 1 edge
The 'edge' channel is now open.
$ snapcraft release httpstat 1 beta
The 'beta' channel is now open.
$ snapcraft release httpstat 1 candidate
The 'candidate' channel is now open.
$ snapcraft release httpstat 1 stable
The 'stable' channel is now open.
$ snapcraft status httpstat
Arch    Channel    Version    Revision
amd64   stable     1.1.3      1
        candidate  1.1.3      1
        beta       1.1.3      1
        edge       1.1.3      1

That’s it!

Install and run the new snap

We are going to uninstall the locally installed httpstat so that we can get it from the Ubuntu Store!

$ snap remove httpstat
httpstat removed
$ snap info httpstat
name:      httpstat
summary:   "Curl statistics made simple"
publisher: simosx
description: |
  httpstat is a utility that analyses show fast is a website
  when you are trying to connect to it.
  This utility is particularly useful to Web administrators
  
channels:              
  stable:    1.1.3 (1) 9MB -
  candidate: 1.1.3 (1) 9MB -
  beta:      1.1.3 (1) 9MB -
  edge:      1.1.3 (1) 9MB -

$ snap install httpstat
httpstat 1.1.3 from 'simosx' installed
$ snap info httpstat
name:      httpstat
summary:   "Curl statistics made simple"
publisher: simosx
description: |
  httpstat is a utility that analyses show fast is a website
  when you are trying to connect to it.
  This utility is particularly useful to Web administrators
  
commands:
  - httpstat
tracking:    stable
installed:   1.1.3 (1) 9MB -
refreshed:   2017-02-04 20:57:31 +0200 EET
channels:              
  stable:    1.1.3 (1) 9MB -
  candidate: 1.1.3 (1) 9MB -
  beta:      1.1.3 (1) 9MB -
  edge:      1.1.3 (1) 9MB -

As an alternative to installing by using the snap command, we can also install the snap in Ubuntu deskop using the Ubuntu Software. Here is how it looks!

Everything looks fine! Let’s finish the tutorial by httpstat-ing ubuntu.com 🙂

$ httpstat https://www.ubuntu.com
HTTP/1.1 200 OK
Date: Sat, 04 Feb 2017 20:23:50 GMT
Server: gunicorn/17.5
Strict-Transport-Security: max-age=15768000
Content-Type: text/html; charset=utf-8
Age: 17
Content-Length: 33671
X-Cache: HIT from privet.canonical.com
X-Cache-Lookup: HIT from privet.canonical.com:80
Via: 1.0 privet.canonical.com:80 (squid/2.7.STABLE7)
Vary: Accept-Encoding

Body stored in: /tmp/tmpybh9xabj

  DNS Lookup   TCP Connection   SSL Handshake   Server Processing   Content Transfer
[    10ms    |      61ms      |     135ms     |       63ms        |       123ms      ]
             |                |               |                   |                  |
    namelookup:10ms           |               |                   |                  |
                        connect:71ms          |                   |                  |
                                    pretransfer:206ms             |                  |
                                                      starttransfer:269ms            |
                                                                                 total:392ms  

2 comments

  • blades

    I can’t find it at the ubuntu software cente (search term: httpstat), though I could install it via snap install…

Leave a Reply

%d bloggers like this: