Let pip Build the Distribution for You

Our wheel script is dandy, but once you need to add more features, it becomes difficult to manage and update. Therefore, Python packaging provides a standard framework, PEP 517, to develop tools that build wheels, so different projects can share those tools and avoid dealing with the details.

There are several PEP 517 backend implementations available. The most famous is Setuptools, but other tools like Flit and Poetry are also gaining traction. You should use them when developing your own project. But let’s roll our own here, because why not!

Note

If you’re wondering: pip is a PEP 517 frontend. So once we have a backend, we can use pip to call it to generate a wheel.

A PEP 517 backend must implement two Python functions:

  • build_wheel
  • build_sdist

Build a wheel

The build_wheel hook can be implemented to, well, build wheels.

packager/pep517.py
import pathlib
import tempfile

from .distinfo import iter_files
from .wheel import create_dist_info, create_wheel


_NAME = "my_package"

_VERSION = "2"

_TAG = "py3-none-any"

_PACKAGE = pathlib.Path("my_package")


def build_wheel(
    wheel_directory, config_settings=None, metadata_directory=None,
):
    with tempfile.TemporaryDirectory() as td:
        if metadata_directory is None:
            td_path = pathlib.Path(td)
            dist_info = create_dist_info(
                _NAME, _VERSION, _TAG, _PACKAGE, td_path,
            )
        else:
            dist_info = pathlib.Path(metadata_directory)

        wheel_path = create_wheel(
            _NAME,
            _VERSION,
            _TAG,
            _PACKAGE,
            dist_info,
            pathlib.Path(wheel_directory),
        )

    return wheel_path.name

Next, we need to tell pip where our backend is. This is done with PEP 518, which introduces the configuration file pyproject.toml, and a TOML table for this:

pyproject.toml
[build-system]
requires = []
backend-path = [""]
build-backend = "packager.pep517"
  • requires lists packages the frontend should install before running the backend. We leave it blank since we include the script in the project, and only use the standard library.
  • backend-path tells the frontend where to find the backend. Empty means where the pyproject.toml file is.
  • build-backend tells the frontend how to import the backend functions.

Now we can let pip build the wheel for us:

$ pip wheel --no-deps /path/to/example-project
Processing /path/to/example-project
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: my-package
  Building wheel for my-package (PEP 517) ... done
  Created wheel for my-package: filename=my_package-2-py3-none-any.whl ...
  Stored in directory: ...
Successfully built my-package

Or we can let pip build and install the wheel in one shot:

$ pip install /path/to/example-project
Processing /path/to/example-project
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: my-package
  Building wheel for my-package (PEP 517) ... done
  Created wheel for my-package: filename=my_package-2-py3-none-any.whl ...
  Stored in directory: ...
Successfully built my-package
Installing collected packages: my-package
  Attempting uninstall: my-package
    Found existing installation: my-package 1
    Uninstalling my-package-1:
      Successfully uninstalled my-package-1
Successfully installed my-package-2

Build a source distribution

A source distribution, a.k.a. sdist, is an archive to destribute the source code with some simple descriptive metadata. It is most commonly used by packages that require binary compilation to support unknown platforms, and is also useful for auditing purposes.

When pip receives an sdist, it first extract the source code, and builds a wheel from it to install. [1] The relation can be drawn like this:

          extract
  +-----------------------+
  |                       v
+-------+  build_sdist  +--------------+
| sdist | <------------ | source code  |
+-------+               +--------------+
                          |
                          | build_wheel
                          v
                        +--------------+
                        |    wheel     |
                        +--------------+
                          |
                          | install
                          v
                        +--------------+
                        | installation |
                        +--------------+
[1]There are historical fallbacks to this, but let’s avoid the details here. The sdist-wheel relation is true in modern contexts.

Since sdists need to be built to make sense anyway, their content is much less rigid than wheels’. The only constraint is basically to always use a gzipped POSIX.1-2001 pax tar with UTF-8 paths.

packager/pep517.py
import tarfile
    return sdist_path.name

Not much we haven’t seen here. This is very similar to how we built wheels, with only slight differences:

  • We use tarfile instead of zipfile to create a gzipped pax tar.
  • We need to bundle both pyproject.toml and the packager directory along, so the sdist can build itself later.

pip does not know how to buld an sdist because it’s a package installer, and the installation process never needs to call build_sdist, as shown in the above graph. But we use the build frontend instead:

$ py -m pip install build
$ py -m build /path/to/example-project --sdist --outdir /path/to/save/wheel

Note

The build tool is still in early development and experiemental. Feel free to report problems to FFY00/python-build.