cmake-pip: pip packages built by CMake

cmake-pip lets pip build and distribute python packages whose extensions are produced by a CMake project. You declare one thing in setup.py – an ExtensionCMake pointing at a CMakeLists.txt and describing it – and pip install / pip wheel does the rest: configure, build, stage the declared artefacts into the package, and make the result relocatable (rpaths, dependency collection, linux soname mangling).

It is designed to be non-invasive on the cmake side: an existing project keeps building exactly as before, with plain cmake; the packaging is a handful of declarations on top. Wheels, source distributions and PEP 660 editable installs all work; dependencies can be provided by conan, entirely driven by cmake-pip.

The shortest possible tour:

# setup.py -- everything else lives in pyproject.toml
from setuptools import setup
from cmake_pip.cmake_extension import ExtensionCMake

setup(ext_modules=[
    ExtensionCMake("my_package.extension1",
                   "my_cmake_project/CMakeLists.txt",
                   cmake_install_component="python_module"),
])

# in my_cmake_project/CMakeLists.txt -- tag what belongs to the package
install(TARGETS extension1
        DESTINATION my_package
        COMPONENT python_module)

then pip install .. The worked examples – each one a fixture of the test suite, built on every CI run – are the recommended starting point.

Getting started

• Worked examples
  • A minimal extension
  • Building only what the wheel needs
  • Running the project’s tests during the build
  • A real dependency from conan-center-index

Guides

• Versioning
  • The two variables
  • Using them in a CMake project
  • Where the version comes from
  • Effect on build tree reuse
• Source distributions
  • What ends up in the sdist
  • The version travels through PKG-INFO
  • Validating with build
• Editable installs
  • What differs from a wheel build
  • The build tree, and why it stays fast
  • Build isolation
  • A single configuration
• Relocation
  • How it runs
  • What is rewritten, per platform
  • Collected dependencies
  • Soname mangling (linux)
• Conan dependencies
  • Declaring the dependencies
  • What runs before the cmake configuration
  • The CONAN_HOME
  • The cmake_pip_default profile
  • Configuration entries and the environment
  • Recipes: vendored, cloned or in place
  • Build tree identity
• Advanced Topics
  • Passing additional options to the CMake build
  • Building only what the package needs
  • Running the test suite

Reference

• Declaring a CMake extension
  • Reference
• CMake commands
  • Package declaration (cmakepip-config.cmake)
  • Relocation (relocation.cmake)
• History

Getting started

Suppose our repository has the following layout:

.
├── setup.py
├── my_cmake_project
|   ├── CMakeLists.txt
|   └── source_code
|       └── python_extension.cpp
└── python_src
    ├── __init__.py
    └── my_pure_python.py

We want a package my_package containing the pure-python module my_pure_python and one extension, extension1, built by the cmake project under my_cmake_project/ (whose internal layout is entirely up to it).

The ExtensionCMake declaration carries exactly three facts:

• the extension being produced: my_package.extension1;

• where its cmake project lives: my_cmake_project/CMakeLists.txt;

• what that project contributes to the package, named through cmake_install_component.

What ends up in the distribution has to be declared: the cmake project tags the install rules of the artefacts belonging to the python package with a COMPONENT, and the extension names that component. After building, cmake-pip stages exactly that component into the distribution with cmake --install --component; the DESTINATION of the install rules is relative to the root of the staged distribution, ie. it is the python package directory:

# in my_cmake_project/CMakeLists.txt
install(TARGETS extension1
        DESTINATION my_package
        COMPONENT python_module)

Declaring a component is barely intrusive for the cmake project: every install() rule already belongs to a component (Unspecified when none is given), a plain cmake --install of the project still installs everything, and tagging many rules at once can be done with a single set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME ...).

This gives, with the static metadata (name, version, packages, authors, …) living in pyproject.toml as usual:

# setup.py -- the plain setuptools setup: cmake-pip installs its build
# commands through a setuptools entry point as soon as an ExtensionCMake is
# declared, no custom cmdclass needed
from setuptools import setup

from cmake_pip.cmake_extension import ExtensionCMake

# "python_module" is the COMPONENT declared on the install rules
ext1 = ExtensionCMake("my_package.extension1",
                      "my_cmake_project/CMakeLists.txt",
                      cmake_install_component="python_module")

setup(ext_modules=[ext1])

Remember to put cmake-pip in the [build-system] requires of pyproject.toml – pip builds run isolated, the backend has to ask for its dependencies. A complete, installable project of exactly this shape opens the worked examples.

Declaring through the cmake-pip cmake module

Instead of writing the install rules by hand, the cmake project can declare its content with the helpers provided by cmake-pip:

find_package(cmakepip REQUIRED)

python_package_add_target(TARGET extension1
                          PYTHON_PACKAGE my_package
                          DESTINATION my_package)

python_package_add_target emits the component install rules (the component is derived from PYTHON_PACKAGE: python_package_MY_PACKAGE) and records the declaration into a small manifest written at the end of the configure phase – cmake_pip_packages.txt in the build tree, plain text, one tab-separated record per line. That manifest is the protocol between the cmake configuration and cmake-pip: thanks to it, setup.py addresses the package simply by its PYTHON_PACKAGE name:

ext1 = ExtensionCMake('my_package.extension1',
                      'my_cmake_project/CMakeLists.txt',
                      cmake_install_component='my_package')

A name that does not appear in the manifest is taken verbatim, which is the plain COMPONENT case of the previous section.

find_package(cmakepip) also declares an umbrella custom target, cmake_pip_target, and every target declared through python_package_add_target becomes a dependency of it. Passing that name as the cmake_target of the extension (the constant CMAKE_PIP_UMBRELLA_TARGET avoids retyping it) restricts the build to exactly the declared extensions – the test executables and benchmarks of the project, although part of all, are not compiled for the wheel:

ext1 = ExtensionCMake('my_package.extension1',
                      'my_cmake_project/CMakeLists.txt',
                      cmake_install_component='my_package',
                      cmake_target=CMAKE_PIP_UMBRELLA_TARGET)

Anything else the package needs built – generated data files, plugins – is attached by the project itself, from any directory, with the regular command:

add_dependencies(cmake_pip_target my_generated_data)

Detecting the cmake-pip modules

cmake-pip ships cmake modules providing the python_package_* helpers. When the ExtensionCMake declares a cmake_install_component, their location is injected into the configuration (cmakepip_DIR) and the regular find_package(cmakepip) resolves them.

A project that must also configure without cmake-pip (a plain C++ build of the same tree) can make the helpers optional by declaring no-op fallbacks:

find_package(cmakepip)
if(NOT cmakepip_FOUND)
  message(STATUS "[CMAKE-PIP] not driven by cmake-pip, python packaging disabled")
  function(python_package_add_target)
  endfunction()
  function(python_package_add_file)
  endfunction()
  function(python_package_create)
  endfunction()
endif()

A project using these fallbacks must also guard its own attachments to the umbrella, which exists only when the module was found:

if(TARGET cmake_pip_target)
  add_dependencies(cmake_pip_target my_generated_data)
endif()

Indices and tables

• Index

• Module Index

• Search Page