Python wheel development

Local setup

Python wheel builds are driven by scikit-build which basically is a setuptools-wrapper for CMake-based projects.

The entry point is setup.py in the root directory. It uses the wheel CMake preset. The preset can be overridden and even other CMake options can be passed via the environment variable CMAKE_ARGS.

You can locally develop and test with the following setup:

# Create a virtual environment inside your source directory
python3 -m venv .venv
# Activate the environment
source .venv/bin/activate
# Install (build) the local Python project
pip install -v .[test]
...
Successfully installed ogs-6.4.2.dev1207

# To build with additional CMake arguments, e.g.:
CMAKE_ARGS="-DOGS_BUILD_PROCESSES=SteadyStateDiffusion" pip install -v .[test]
# OR
pip install -v .[test] --config-settings=cmake.args="-DOGS_BUILD_PROCESSES=SteadyStateDiffusion"

The pip install-step starts a new CMake-based ogs build in _skbuild-subdirectory (inside the source code) using the wheel-preset. When the CMake build is done it installs the wheel into the activated virtual environment and you can interact with it.

The contents of _skbuild/[platform-specific]/cmake-install will make up the wheel.

Testing

# Run python tests
pytest
============================================== test session starts ===============================================
platform darwin -- Python 3.10.6, pytest-7.1.3, pluggy-1.0.0
rootdir: ~/code/ogs/ogs, configfile: pyproject.toml, testpaths: Tests
collected 2 items

Tests/Python/test_cli.py .                                                                                 [ 50%]
Tests/Python/test_simlator.py .                                                                            [100%]

=============================================== 2 passed in 0.55s ================================================

# Start the python interpreter
python3
>>> import ogs.simulator as sim
>>> sim.initialize(["", "--help"])

If you make modifications on the C++ side you need to run pip install .[test] again. Modifications on the Python tests are immediately available to pytest.

To get the output of a specific test:

pytest --capture=tee-sys ./Tests/Python/test_simulator_mesh_interface.py

Module structure

Python modules added in CMake via pybind11_add_module() should only contain the binding code and other (helper code) which is used by that module only! If you need helper code which is used by several modules (e.g. OGSMesh-class used in mesh- and simulator-module) it needs to be defined in a regular library and linked to the modules.

If you don’t do this you will get unresolved externals between the modules and when you try to link them together you will get runtime errors or it doesn’t link at all (at least on Windows, because of the .pyd library format).

CI

For generating the various wheels for different Python versions and platforms cibuildwheel is used.

You can test it locally with, e.g. only building for Python 3.10:

CIBW_BUILD="cp310*" pipx run cibuildwheel

Please note that on Linux cibuildwheel runs the builds inside manylinux Docker containers. On other platforms the build happens with native tools. See the cibuildwheel docs for more information.

Wheels are generated in the wheelhouse/-folder.

cibuildwheel is configured in pyproject.toml:

[tool.cibuildwheel]
...

This article was written by Lars Bilke. If you are missing something or you find an error please let us know.
Generated with Hugo 0.122.0 in CI job 425680 | Last revision: March 20, 2023
Commit: [PL/THM] Add I/F for internal state variables I/O 24b0455  | Edit this page on