Nuxi The CloudABI Development Blog

Welcoming all Python enthusiasts: CPython 3.6 for CloudABI!

August 1, 2016 by Ed Schouten

Late last year, I gave a talk about CloudABI at 32C3 in Hamburg. Not long after the conference, I was contacted by Alex Willmer. Being a Python enthusiast, Alex told me he wanted to help me complete the port of CPython to CloudABI. Thanks to his work, we now have initial support for running Python scripts in a strongly sandboxed, yet easy to understand environment. In addition to that, he helped me spread the word by giving a talk about CloudABI at EuroPython. In today’s article, let’s take a look at how Python for CloudABI works!

Importing modules

One of the prerequisites of having a usable Python environment is that scripts are capable of importing modules. Python modules come in at least three different flavours:

An interesting observation is that many of the core modules written in C are never imported directly. For example, the socket module is written in Python entirely. In order to create sockets at the operating system level, it makes use of an internal _socket module, which is written in C. This means that being able to load modules written in Python is crucial, as they are needed to get even the simplest programs to work.

As CloudABI makes use of capability-based security (Capsicum), it is not allowed to open files by absolute path. Files may only be opened as pairs of a directory file descriptor and a relative pathname string, using openat(). This requires Python’s module loader, importlib, to be patched up, as it currently searches for its modules by concatenating them to the pathname strings stored in sys.path.

Fortunately, Python is one of the few languages that provides very good support for file system operations relative to directory file descriptors. As of Python 3.3, all of the file system functions provided by os have an additional dir_fd argument. When provided, the *at() functions are invoked instead. To make module loading work, Alex has extended importlib to make use of this feature. sys.path may now contain directory file descriptors as well.

Interacting with the outside world

An important aspect of CloudABI is that processes are started with the right set of file descriptors, as these determine what a process may and may not do during its lifetime. To make this easy to set up, CloudABI programs are configured through a YAML file to declare configuration parameters and resource dependencies, so that they can be started by cloudabi-run. This utility acquires the resources specified in the YAML file, so that CloudABI processes can be started with sandboxing turned on immediately.

Our version of Python also integrates with this model seamlessly by propagating a subtree of the YAML configuration file directly to the Python script. It is automatically converted to native Python data types (strings, integers, dictionaries, lists, etc) and exposed under the name sys.argdata. In the next couple of examples, you’ll see how this works in practice.

Example 1: a sandboxed ‘Hello World’

Below is a pretty simple YAML configuration file that can be used to start Python in such a way that it prints a formatted message to the console. During its lifetime, it only has access to stdout, stderr and the Python modules directory.

%TAG ! tag:nuxi.nl,2015:cloudabi/
---
stderr: !fd stderr               # Let Python write backtraces to stderr.
path:                            # Search paths for Python modules.
    - !file
        path: /usr/local/x86_64-unknown-cloudabi/lib/python3.6
args:                            # Arguments exposed as sys.argdata.
    first_name: Konrad
    last_name: Zuse
    output: !fd stdout
command: |                       # Inline Python script, like 'python -c'.
    import io
    import sys

    argdata = sys.argdata
    s = io.TextIOWrapper(argdata['output'], encoding='UTF-8')
    s.write('Hello, %s %s!\n' % (argdata['first_name'], argdata['last_name']))

The following commands show how Python can be installed and run with this configuration file on a FreeBSD system.

$ sudo pkg install x86_64-unknown-cloudabi-python
$ cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < hello.yaml
Hello, Konrad Zuse!

Example 2: a sandboxed web server

The second example is of a simple web server that is capable of serving fixed HTTP responses. During its lifetime, it only has access to an already bound TCP socket, stderr and the Python modules directory.

In this example, the Server class is similar to Python’s TCPServer, except that it allows the socket object to be injected. The HTTPRequestHandler class is a simple wrapper around Python’s BaseHTTPRequestHandler, implementing the HTTP response that needs to be returned.

%TAG ! tag:nuxi.nl,2015:cloudabi/
---
stderr: !fd stderr               # Let Python write backtraces to stderr.
path:                            # Search paths for Python modules.
    - !file
        path: /usr/local/x86_64-unknown-cloudabi/lib/python3.6
args:                            # Arguments exposed as sys.argdata.
    my_socket: !socket
        bind: 0.0.0.0:12345
command: |                       # Inline Python script, like 'python -c'.
    import http.server
    import socketserver
    import sys

    class Server(socketserver.BaseServer):
        def __init__(self, socket, RequestHandlerClass):
            socketserver.BaseServer.__init__(self, None, RequestHandlerClass)
            self.socket = socket

        def fileno(self):
            return self.socket.fileno()

        def get_request(self):
            return self.socket.accept()

    class HTTPRequestHandler(http.server.BaseHTTPRequestHandler):
        def do_GET(self):
            self.send_response(200)
            self.send_header('Content-Type', 'text/html')
            self.end_headers()
            self.wfile.write(b'<marquee>It works!</marquee>')

    if __name__ == '__main__':
        Server(sys.argdata['my_socket'], HTTPRequestHandler).serve_forever()

Where to go from here?

It’s amazing to see how much functionality we already managed to get working. That said, there are quite a lot of things that still need to be done. To be able to get Python to work, we had to write a fair number of patches. Many of these patches need to be polished up, so we can send them upstream. It would be nice if most of that work is finished before Python 3.7 comes out.

In the meantime, we do invite people to play around with the version of Python that is already there and share their experiences, either on IRC (#cloudabi on EFnet) or on our mailing list (cloudabi-devel@googlegroups.com). Enjoy!