Workflow testing

Introduction

We use the workflow manager Snakemake to test workflows which consist of the execution of several steps which are based on each other.

Requirements

• Snakemake >= 7.3.0
  • If you use OGS_USE_PIP=ON then snakemake is installed in your virtual environment in your build-directory automatically. You can then call it via .venv/bin/snakemake ... (inside your build directory).
• On Windows only:
  • The tee-utility in the PATH (can be installed from https://sourceforge.net/projects/unxutils)

Examples

• ExtractBoundary.smk
• VoxelGridFromLayers.smk

These example workflows are added to CTest as well:

if(SNAKEMAKE AND NOT OGS_USE_MPI)
    add_test(NAME snakemake_ExtractBoundary
        COMMAND ${SNAKEMAKE} -j 1
            --configfile ${PROJECT_BINARY_DIR}/buildinfo.yaml
            -s ${CMAKE_CURRENT_SOURCE_DIR}/ExtractBoundary.smk
    )
    add_test(NAME snakemake_VoxelGridFromLayers
    # ...
    )
    add_dependencies(ctest ExtractBoundary Layers2Grid AddFaultToVoxelGrid)
endif()

Modularization

We started on implementing modular rule definitions and tool wrapper in scripts/snakemake.

Links

• Snakemake Documentation
• Short tutorial
• Software Carpentry Workshop
• HPC Carpentry Workshop with Snakemake