Extending Cython with GIL-free types

Bypass the GIL with thread-safe Python-like objects

Xavier Thompson

C-Languages Compiler and Interpreters Cython Open-Source

See in schedule

Cython is a compiled language, largely inspired by Python, which can already provide both massive speedups and seamless interoperability with C libraries. In Cython, Python and C can freely mix thanks to a unified syntax and gradual typing. This opens the door to bypassing the GIL by restricting oneself to the C side of Cython, but it entails letting go of all the high-level semantics of Python and going back to low-level manual memory management, pointer arithmetic and good old segmentation faults... Not to mention all the joys of multi-threading in C!

In this talk, we will recount how we experimentally extended Cython to bridge that gap with GIL-free reference-counted types and a type system for thread-safe concurrency. We call this extension: Cython+.

By hacking a bit the Cython compiler, one can create new language features which translate under-the-hood to low-level GIL-agnostic C or C++. In this way we extended Cython with a new class system that looks familiar to a Python programmer, but can be used in Cython while the GIL is released because it does not depend on the Python/C API. By adapting the way Cython generates reference counting instructions for Python objects, we equipped our types with a thread-safe reference count. Thanks to Cython's expertise with the Python/C API, we wrapped our types into extension types, thus gaining interoperability with Python. And using Cython's static type checking, we are currently introducing an ownership type system for thread-safe concurrency inspired by Pony and Rust.

Type: Talk (30 mins); Python level: Beginner; Domain level: Intermediate


Xavier Thompson

Nexedi

Xavier embarked on the Cython+ project as an intern at Nexedi. Before that, he was studying embedded systems at Telecom Paris University. He likes bare-metal embedded systems and loves discovering new things about language design and type systems. He shares his time between his hometown in the foothills of the Pyrenees mountains and his work life in northern Lille. Back when the world was still sane, he enjoyed acting and formal debating. This will be his first talk ever at a conference.