The executable OGS for MPI parallel computing is compiled with special build configuration. If you need to compile the source code, please read [Build configuration for MPI and PETSc](https://www.opengeosys.org/docs/devguide/getting-started/build-configuration_for_mpi_petsc/configure_for_mpi_and_petsc/
To conduct DDC enabled parallel computing with OGS, following steps are required:
For the domain decomposition approach, an application of OGS using METIS as a node wise
mesh topology partitioner,
partmesh, is provided to partition meshes by node. An example of how
to partition meshes is available in a workflow documentation on how to create a simple parallel model. You can type
partmesh --help for the detailed
command line options.
The partitioned meshes are written in binary files for a fast parallel reading in OGS.
Setting PETSc solver is the only change in project file for running parallel OGS with PETSc.
For linear solver, it is done by adding a tag of
<linear_solvers> <linear_solver> <name>linear_solver</name> <eigen> <solver_type>SparseLU</solver_type> <scaling>true</scaling> </eigen> <petsc> <parameters>-ksp_type bcgs -pc_type jacobi -ksp_rtol 1.e-16 -ksp_atol 1.e-12 -ksp_max_it 4000 </parameters> </petsc> </linear_solver> </linear_solvers>
If tag of
petsc is not given in project file, the default setting of PETSc
linear solver will be taken, which uses the solver type of
the preconditioner type of
For the simulation of coupled processes with the staggered scheme, a prefix
can be used by tag
prefix to set PETSc solver for each process, individually.
<linear_solvers> <linear_solver> <name>linear_solver_T</name> <petsc> <prefix>T</prefix> <parameters>-T_ksp_type bcgs -T_pc_type bjacobi -T_ksp_rtol 1e-16 -T_ksp_max_it 3000 </parameters> </petsc> </linear_solver> <linear_solver> <name>linear_solver_H</name> <petsc> <prefix>H</prefix> <parameters>-H_ksp_type bcgs -H_pc_type mg -H_ksp_rtol 1e-12 -H_ksp_max_it 4000 </parameters> </petsc> </linear_solver> </linear_solvers>
The above example shows that once a prefix is given for PETSc linear solver
settings, the original prefix of PETSc
-, can be replaced with a new prefix,
-[given prefix string]_. In the
-, is replaced with
A introduction and a list of PETSc KSP solvers and preconditioners can be found by this link.
For MPI launcher, either
mpirun can be used to run OGS.
mpiexec is recommended because it is defined in the MPI standard.
The number of processes to run of
mpiexec must be identical to the number of
For example, if the meshes of a project,
foo.prj, are partitioned into 5 partitions,
OGS can be launched in MPI as
mpiexec -n 5 ogs foo.prj -o [path to the output directory]
Running PETSc enabled OGS with one compute thread does not need mesh partitioning.
However, the MPI launcher
mpirun is required, e.g.:
mpiexec -n 1 ogs ...
Additional PETSc command line options can be given as unlabelled arguments at the end of the OGS run command preceded by two minus-signs
... ogs ... -- [PETSc options]).
With PETSc command line options, you can
mpiexec -n 5 ogs foo.prj -o output -- -ksp_converged_reason -ksp_monitor_true_residual
mpiexec -n 5 ogs foo.prj -o output -- -ksp_type gmres -ksp_rtol 1e-16 -ksp_max_it 2000
For Linux clusters or supercomputer, computation job has to be submitted to queue and job management system, which may provide a special command to launch MPI job. A job script for such queue system is required.
The cluster system EVE of UFZ uses SLURM (Simple Linux Utility for Resource Management) to manage computing jobs. Here is an example of job script for the SLURM system on EVE:
#!/bin/bash #SBATCH --job-name=ARESH_HM_3D_20 #SBATCH --chdir=/home/wwang/data_D/project/AREHS/HM_3D #SBATCH --output=/home/wwang/data_D/project/AREHS/HM_3D/output/log_%x_%j.txt #SBATCH --error=/home/wwang/data_D/project/AREHS/HM_3D/output/err_%x_%j.txt #SBATCH --time=0-48:00:00 #SBATCH -n 20 #SBATCH --mem-per-cpu=4G #SBATCH --email@example.com #SBATCH --mail-type=BEGIN,END ###source /etc/profile.d/000-modules.sh export MODULEPATH="/software/easybuild-broadwell/modules/all/Core:/software/modulefiles" module load foss/2020b petsc-bilke/3.16.5_foss2020b module load OpenMPI/4.0.5 HDF5/1.10.7 GMP/6.2.0 APP="/home/wwang/code/ogs6/exe_petsc/bin/ogs" /bin/echo In directory: `pwd` /bin/echo Number of CPUs: $SLURM_CPUS_PER_TASK /bin/echo File name: $1 ##load $APP srun -n "$SLURM_NTASKS" $APP /home/wwang/data_D/project/AREHS/HM_3D/simHM_glaciation.prj -o /home/wwang/data_D/project/AREHS/HM_3D/output
In the job script for EVE,
module load foss/2020b must be presented, and
srun is a sort of MPI job launcher.
If a job fails with an error message about shared library not found, you can check
the EVE modules specified in the files in source code directory:
scripts/env/eve, and add the corresponding modules to the load list
in the job script.
Once the job script is ready, you can
sbatch [job script name],
scancel [job ID].
For the detailed syntax of job script of SLURM for EVE, please visit https://wiki.ufz.de/eve/ (user login required).
There are two output types available, VTK and XMDF.
In the project file, output type can be specified in
type tag of
<output> <type>VTK</type> ... </output>
<output> <type>XDMF</type> ... </output>
The results are output in the partitioned VTU files, which are governed by a PVTU file. The data in the ghost cells of VTU files are overlapped. An OGS utility, RemoveGhostData, is available to merge the partition VTU files into one VTU file, meanwhile to eliminate the data overlapping. Here is an example to use that utility:
RemoveGhostData -i foo.pvtu -o foo.vtu
Where the input file name is the name of PVTU file.
With XMDF, OGS outputs two files, one XMDF file and one HDF5 file with file name
h5. You can use ParaView to open the XDMF file by selecting
Xdmf3ReaderT. The XMDF output is highly recommended for
running OGS with a large mesh, especially on supercomputers.