Icon Rufen Sie uns an
+49 441.309197-69 +49 441.309197-69
 
EN

Packaging python applications

Posted by Ruben Schuller on Monday, December 04, 2017

Various tools and applications used by bytemine are written in Python. During the development of the CrashPlan Pro Icinga check, the question arose how to make it deployable reasonably hassle-free. Following is described a way to package a small python application which fetches our home page into one executable file which can be moved around systems.

Clean environments with virtualenv

We will be using virtualenv for development. It creates a local python environment, so we don't have to clutter our global python installation with dependencies:

$ virtualenv venv             # create a new virtualenv, stored in the "venv" subfolder
$ . venv/bin/activate         # activate virtualenv by sourcing the activation script
(venv) $ pip install requests # install requests package in the virtualenv
(venv) $ python -c 'import requests; print requests.get("https://bytemine.net").text'
<!DOCTYPE html>
<html class="no-js">
...

To record the packages installed in the virtualenv, we can use:

(venv) $ pip freeze > requirements.txt

and install them somewhere else using:

(another-venv) $ pip install -r requirements.txt

Adding setup.py

To make a module distributable, the pythonic way is to write a setup.py script using setuptools. This is also required for uploading to PyPI.

The files of the module have to be in a subdirectory of the path containing setup.py, together with an empty file __init__.py to let python know that this directory is an importable module:

$ ls -1 *
setup.py

mymodule:
__init__.py
mymodule.py

$ cat mymodule/mymodule.py
import requests
def main():
    print requests.get("https://bytemine.net").text
if __name__=="__main__":
    main()

setup.py contains information about the author, depencencies, etc. If the described module is a script which can be executed, an entry point can also be described for which the setup will create a command to run it (e.g. $VIRTUAL_ENV/bin/mymodule if installed in a virtualenv):

from setuptools import setup

setup(
    name="mymodule",
    version="0.0.1",
    description="My small testing module",
    packages=["mymodule"],
    author="bytemine GmbH",
    author_email="schuller@bytemine.net",
    install_requires=[
        "certifi==2017.11.5",
        "chardet==3.0.4",
        "idna==2.6",
        "requests>=2.18.4",
        "urllib3==1.22"
    ],
    entry_points={'console_scripts':['mymodule=mymodule.mymodule:main']}
)

Afterwards we can execute python setup.py install to install our package into the active python environment. The requirements are usually the same as reported by pip freeze, but optionally allow for more complicated setups.

Installing it somewhere

Now we have a python package which can be installed using pythons setuptools. One way to install our package would be to create a tarball of the sources using python setup.py sdist, copy it somewhere and install it - possibly into a virtualenv - with python setup.py install. But this requires activating the virtualenv each time we want to run it:

$ . venv/bin/activate && mymodule

This works, but isn't great. Each time we want to install it we have to create a virtualenv, either manually or using a deployment script.

Packing it up with pex

pex enables packaging a python application together with its depencencies into one executeable file, which only requires a python interpreter to run.

Following the official documentation, we can install pex like this, installing it to $HOME/bin:

$ virtualenv pex-venv
$ . pex-venv/bin/activate
(pex-venv) $ pip install pex
(pex-venv) $ pex pex requests -c pex -o ~/bin/pex
(pex-venv) $ deactivate
$ PATH=$HOME/bin:$PATH && export PATH
$ pex --version                                                                                                                                                                                       
pex 1.2.13

To package our application we run pex like the following, "." requests that the current directory should be included in the pex file (using our setup.py):

$ pex . --disable-cache -m mymodule.mymodule -o mymodule.pex

During development, the switch --disable-cache should be used, otherwise pex would not include changes in our source code as long as version in setup.py isn't changed.

The resulting file mymodule.pex can now be used with python interpreters of the same version (2 or 3) and architecture (because of python C-extensions) is available:

$ ./mymodule.pex
<!DOCTYPE html>
<html class="no-js">
...

Take a look at the pex documentation for how to package for other environments. With a recently introduced feature, you can even build pex files which support multiple platforms and interpreter versions (though I haven't tested this yet!).

Packaging the package

At bytemine, software is usually deployed using the packaging mechanism of the distribution used. To create such packages we use fpm-cookery. Returning to the beginning of this post, this is the recipe used to package the CrashPlan Pro Icinga check:

class CPPAlertCheck < FPM::Cookery::Recipe
    description 'bytemine CrashPlan Pro alert check for icinga'
    name     'bytemine-cpp-alert-check'
    version  '0.0.2'
    vendor   'bytemine'
    revision  1
    homepage 'https://www.bytemine.net/'
    source   "https://files.bytemine.net/#{name}-#{version}.tgz"
    sha256   'cab3a90c4f30f774dd16acd60f764210008b7c56d7195bc565038f54cbbc545b'
    arch     'x86_64'
    maintainer 'bytemine GmbH <support@bytemine.net>'
    depends 'python2.7'

    def build
    end

    def install
        share(name).install "cpp-alert-check.pex"
        share(name).install "README.md"
    end
end

With fpm-cookery installed we can now run

$ fpm-cook

and have our package dropped in pkg/bytemine-cpp-alert-check_0.0.2-1-amd64.deb. The type of package is by default set to match the system you run fpm-cookery on. To build for other distributions the -t TARGET switch can be used, for example fpm-cook -t rpm.

Conclusion

Having the possibility to have full python applications stored in one file is a handy tool to easily run it somewhere else. As all depencencies are included, the resulting pex file can also be packaged for distributions, regardless which version of python modules they ship.