Why should I use scikit-build ?¶
Scikit-build is a replacement for distutils.core.Extension with the following advantages:
provide better support for additional compilers and build systems
first-class cross-compilation support
location of dependencies and their associated build requirements
Basic Usage¶
Example of setup.py, CMakeLists.txt and pyproject.toml¶
To use scikit-build in a project, place the following in your project’s setup.py file:
from skbuild import setup # This line replaces 'from setuptools import setup'
Your project now uses scikit-build instead of setuptools.
Next, add a CMakeLists.txt
to describe how to build your extension. In the following example,
a C++ extension named _hello
is built:
cmake_minimum_required(VERSION 3.11.0)
project(hello)
find_package(PythonExtensions REQUIRED)
add_library(_hello MODULE hello/_hello.cxx)
python_extension_module(_hello)
install(TARGETS _hello LIBRARY DESTINATION hello)
Then, add a pyproject.toml
to list the build system requirements:
[build-system]
requires = ["setuptools", "wheel", "scikit-build", "cmake", "ninja"]
Note
By default, scikit-build looks in the project top-level directory for a
file named CMakeLists.txt
. It will then invoke cmake
executable
specifying a generator matching the python being used.
Setup options¶
setuptools options¶
The section below documents some of the options accepted by the setup()
function.
packages
: Explicitly list of all packages to include in the distribution. Setuptools will not recursively scan the source tree looking for any directory with an__init__.py
file. To automatically generate the list of packages, see Using find_package().package_dir
: A mapping of package to directory namesinclude_package_data
: If set toTrue
, this tells setuptools to automatically include any data files it finds inside your package directories that are specified by yourMANIFEST.in
file. For more information, see the setuptools documentation section on Including Data Files.package_data
: A dictionary mapping package names to lists of glob patterns. For a complete description and examples, see the setuptools documentation section on Including Data Files. You do not need to use this option if you are using include_package_data, unless you need to add e.g. files that are generated by your setup script and build process. (And are therefore not in source control or are files that you don’t want to include in your source distribution.)exclude_package_data
: Dictionary mapping package names to lists of glob patterns that should be excluded from the package directories. You can use this to trim back any excess files included by include_package_data. For a complete description and examples, see the setuptools documentation section on Including Data Files.py_modules
: List all modules rather than listing packages. More details in the Listing individual modules section of the distutils documentation.data_files
: Sequence of (directory, files) pairs. Each (directory, files) pair in the sequence specifies the installation directory and the files to install there. More details in the Installing Additional Files section of the setuptools documentation.entry_points
: A dictionary mapping entry point group names to strings or lists of strings defining the entry points. Entry points are used to support dynamic discovery of services or plugins provided by a project. See Dynamic Discovery of Services and Plugins for details and examples of the format of this argument. In addition, this keyword is used to support Automatic Script Creation.scripts
: List of python script relative paths. If the first line of the script starts with#!
and contains the word python, the Distutils will adjust the first line to refer to the current interpreter location. More details in the Installing Scripts section of the distutils documentation.
New in version 0.8.0.
zip_safe
: A boolean indicating if the Python packages may be run directly from a zip file. If not already set, scikit-build sets this option toFalse
. See Setting the zip_safe flag section of the setuptools documentation.
Note
As specified in the Wheel documentation, the universal
and python-tag
options
have no effect.
scikit-build options¶
Scikit-build augments the setup()
function with the following options:
cmake_args
: List of CMake options.
For example:
setup(
[...]
cmake_args=['-DSOME_FEATURE:BOOL=OFF']
[...]
)
cmake_install_dir
: relative directory where the CMake artifacts are installed. By default, it is set to an empty string.cmake_source_dir
: Relative directory containing the projectCMakeLists.txt
. By default, it is set to the top-level directory wheresetup.py
is found.``cmake_manifest_process_hook`: Python function consumming the list of files to be installed produced by cmake. For example, cmake_manifest_process_hook can be used to exclude static libraries from the built wheel.
For example:
def exclude_static_libraries(cmake_manifest):
return list(filter(lambda name: not (name.endswith('.a')), cmake_manifest))
setup(
[...]
cmake_process_manifest_hook=exclude_static_libraries
[...]
)
New in version 0.5.0.
cmake_with_sdist
: Boolean indicating if CMake should be executed when running sdist command. Setting this option toTrue
is useful when part of the sources specified inMANIFEST.in
are downloaded by CMake. By default, this option isFalse
.
New in version 0.7.0.
cmake_languages
: Tuple of languages that the project use, by default (‘C’, ‘CXX’,). This option ensures that a generator is chosen that supports all languages for the project.cmake_minimum_required_version
: String identifying the minimum version of CMake required to configure the project.
Scikit-build changes the following options:
New in version 0.7.0.
setup_requires
: Ifcmake
is found in the list, it is explicitly installed first by scikit-build.
Command line options¶
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] [skbuild_opts] [cmake_configure_opts] [-- [cmake_opts] [-- [build_tool_opts]]]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
There are few types of options:
-
[global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
--help [cmd1 cmd2 ...]
cmd --help
scikit-build options:
[skbuild_opts]
CMake configure options:
[cmake_configure_opts]
CMake options:
[cmake_opts]
build tool options:
[build_tool_opts]
setuptools, scikit-build and CMake configure options can be passed normally, the cmake and
build_tool set of options needs to be separated by --
:
Arguments following a "--" are passed directly to CMake (e.g. -DSOME_FEATURE:BOOL=ON).
Arguments following a second "--" are passed directly to the build tool.
setuptools options¶
For more details, see the official documentation.
scikit-build extends the global set of setuptools options with:
New in version 0.4.0.
Global options:
[...]
--hide-listing do not display list of files being included in the
distribution
New in version 0.5.0.
Global options:
[...]
--force-cmake always run CMake
--skip-cmake do not run CMake
Note
As specified in the Wheel documentation, the --universal
and --python-tag
options
have no effect.
scikit-build options¶
scikit-build options:
--build-type specify the CMake build type (e.g. Debug or Release)
-G , --generator specify the CMake build system generator
-j N allow N build jobs at once
[...]
New in version 0.7.0.
scikit-build options:
[...]
--cmake-executable specify the path to the cmake executable
New in version 0.8.0.
scikit-build options:
[...]
--skip-generator-test skip generator test when a generator is explicitly selected using --generator
CMake Configure options¶
New in version 0.10.1.
These options are relevant when configuring a project and can be passed as global options using setup.py or pip install.
The CMake options accepted as global options are any of the following:
-C<initial-cache> = Pre-load a script to populate the cache.
-D<var>[:<type>]=<value> = Create or update a cmake cache entry.
Warning
The CMake configure option should be passed without spaces. For example, use -DSOME_FEATURE:BOOL=ON instead of -D SOME_FEATURE:BOOL=ON.
CMake options¶
These are any specific to CMake. See list of CMake options.
For example:
-DSOME_FEATURE:BOOL=OFF
build tool options¶
These are specific to the underlying build tool (e.g msbuild.exe, make, ninja).
Advanced Usage¶
How to test if scikit-build is driving the compilation ?¶
To support the case of code base being built as both a standalone project
and a python wheel, it is possible to test for the variable SKBUILD
:
if(SKBUILD)
message(STATUS "The project is built using scikit-build")
endif()
Adding cmake as building requirement only if not installed or too low a version¶
If systematically installing cmake wheel is not desired, the setup_requires
list
can be set using the following approach:
from packaging.version import LegacyVersion
from skbuild.exceptions import SKBuildError
from skbuild.cmaker import get_cmake_version
# Add CMake as a build requirement if cmake is not installed or is too low a version
setup_requires = []
try:
if LegacyVersion(get_cmake_version()) < LegacyVersion("3.4"):
setup_requires.append('cmake')
except SKBuildError:
setup_requires.append('cmake')
Enabling parallel build¶
Ninja¶
If Ninja generator is used, the associated build tool (called ninja
)
will automatically parallelize the build based on the number of available CPUs.
To limit the number of parallel jobs, the build tool option -j N
can be passed
to ninja
.
For example, to limit the number of parallel jobs to 3, the following could be done:
python setup.py bdist_wheel -- -- -j3
For complex projects where more granularity is required, it is also possible to limit the number of simultaneous link jobs, or compile jobs, or both.
Indeed, starting with CMake 3.11, it is possible to configure the project with these options:
For example, to have at most 5 compile jobs and 2 link jobs, the following could be done:
python setup.py bdist_wheel -- \
-DCMAKE_JOB_POOL_COMPILE:STRING=compile \
-DCMAKE_JOB_POOL_LINK:STRING=link \
'-DCMAKE_JOB_POOLS:STRING=compile=5;link=2'
Unix Makefiles¶
If Unix Makefiles generator is used, the associated build tool (called make
)
will NOT automatically parallelize the build, the user has to explicitly pass
option like -j N
.
For example, to limit the number of parallel jobs to 3, the following could be done:
python setup.py bdist_wheel -- -- -j3
Visual Studio IDE¶
If Visual Studio IDE generator is used, there are two types of parallelism:
target level parallelism
object level parallelism
Warning
Since finding the right combination of parallelism can be challenging, whenever possible we recommend to use the Ninja generator.
To adjust the object level parallelism, the compiler flag /MP[processMax]
could
be specified. To learn more, read /MP (Build with Multiple Processes).
For example:
set CXXFLAGS=/MP4
python setup.py bdist_wheel
Starting with Visual Studio 2010, the target level parallelism can be set from command line
using /maxcpucount:N
. This defines the number of simultaneous MSBuild.exe
processes.
To learn more, read Building Multiple Projects in Parallel with MSBuild.
For example:
python setup.py bdist_wheel -- -- /maxcpucount:4
Support for isolated build¶
New in version 0.8.0.
As specified in PEP 518, dependencies required at install time can be specified using a
pyproject.toml
file. Starting with pip 10.0, pip reads the pyproject.toml
file and
installs the associated dependencies in an isolated environment. See the pip build system interface
documentation.
An isolated environment will be created when using pip to install packages directly from source or to create an editable installation.
scikit-build supports these use cases as well as the case where the isolated environment support
is explicitly disabled using the pip option --no-build-isolation
available with the install,
download and wheel commands.
Optimized incremental build¶
To optimize the developer workflow, scikit-build reconfigures the CMake project only when needed. It caches the environment associated with the generator as well as the CMake execution properties.
The CMake properties are saved in a CMake spec file
responsible
to store the CMake executable path, the CMake configuration arguments, the CMake version as well as the
environment variables PYTHONNOUSERSITE
and PYTHONPATH
.
If there are no CMakeCache.txt
file or if any of the CMake properties changes, scikit-build will
explicitly reconfigure the project calling skbuild.cmaker.CMaker.configure()
.
If a file is added to the CMake build system by updating one of the CMakeLists.txt
file, scikit-build
will not explicitly reconfigure the project. Instead, the generated build-system will automatically
detect the change and reconfigure the project after skbuild.cmaker.CMaker.make()
is called.
Cross-compilation¶
See CMake Toolchains.
Examples for scikit-build developers¶
Note
To be documented. See #227.
Provide small, self-contained setup function calls for (at least) two use cases:
when a CMakeLists.txt file already exists
when a user wants scikit-build to create a CMakeLists.txt file based on the user specifying some input files.