Building Cython (or C) extensions using uv

#cython #open-source #programming #python

Building and maintaining Python libraries with extension modules can be somewhat tricky. I maintain one such library called streaming-form-data, that provides a streaming parser for multipart/form-data input and has all the performance-critical code written using Cython.

In this post I'll share my experiences transitioning this library from Poetry to uv. If you also maintain such a library or are just plain curious about how uv can work together with Cython, you may find this post useful.

Why transition to uv?

For years, I've been using Poetry for streaming-form-data, together with a custom build.py script for handling the C extension. While this setup is not explicitly documented / supported by Poetry, it has worked really well so far.

Last week though, the release of Poetry 2.0.0 introduced a bug that broke my build process, and introduced installation errors for some users of the library. After spending some time debugging the issue, I realized that this issue would be best fixed upstream (at Poetry's end). At the same time I've been looking into uv with curiosity for a few months now, as I'm sure most of the members in the Python community have been. I wasn't actively looking to move away from Poetry, but this seemed like a good opportunity to try uv out!

The move went pretty well! I published version 1.19.1 of the package yesterday which was the first version built using uv!

uv and build backends

uv is the latest tool in the Python ecosystem that has taken almost the entire community by storm. It's extremely fast. It consolidates multiple tools into one. It can manage the installation of multiple Python distributions for you. And did I mention that it's fast?

uv also allows building and publishing packages to PyPI, delegating the actual package building to something called a "build backend". Popular build backend choices at the time of this writing include hatchling (ideal for pure Python projects), setuptools (versatile and battle-tested choice that supports C extensions), flit (focused on simplicity for pure Python projects), and a few others.

setuptools remains a solid choice for projects that use Cython or include a pure C extension. Despite misconceptions about its status (no, it's not deprecated), setuptools continues to evolve and is actively maintained by the PyPA.

It's also what I ended up going with for streaming-form-data.

Cython

Cython is a superset of Python that gives you the ability to compile your Python code into optimized C code, which can then be included as a normal C extension into your library. There are several ways of distributing Cython code in libraries. I personally use the following approach:

  1. Compile the .pyx source code into .c using the cython CLI.
  2. Include the generated .c file into the source distribution, so that end users don't need to have Cython installed.

I've been using this approach for quite a few years now and haven't had any reasons to complain so far.

pyproject.toml configuration

Here's how I configured all the above in the project's pyproject.toml:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
package-dir = {"" = "src"}
ext-modules = [
    {name = "streaming_form_data._parser", sources = ["src/streaming_form_data/_parser.c"]}
]

The build-system section defines what build backend to use. package-dir in tool.setuptools maps the package directory to src, aligning with modern Python packaging recommendations. And finally the ext-modules list defines the C extension to be built, using the .c file generated by Cython.

Conclusion

In my case, the transition to uv has worked quite well!

One standout feature is its remarkable speed. For instance, in my Github Actions pipeline, the step for installing the package dependencies now takes only 7 seconds, down from 14 earlier. Resolving packages locally is almost instantaneous, and the entire experience is incredibly snappy. What I appreciate most, though, is the simplicity of managing just one tool (uv) instead of juggling both pyenv and Poetry.

Beyond its performance, uv is rapidly gaining traction in the Python community. While I typically wait a few months before switching to the shiny new object, this time, the switch has been worth it. If you're considering making the move, uv is a great choice!