Create a simple parallel model

Unsupported Processes

LIE related processes are not supported to use the parallel FEM scheme with PETSc.

Software Modules and Compilation on EVE

• Checkout the OGS-6 source code
  • Source the environment scripts scripts/env/eve/cli.sh
• Configure with utilities enabled to:
  • Create the binary for generation of the structured mesh
  • Create the binary for the mesh partitioner tool partmesh
  • cmake <path_to_source> -DOGS_BUILD_UTILS=ON
• Configure OGS for parallel usage with PETSc:
  • Source the environment scripts scripts/env/eve/petsc.sh
  • cmake <path_to_source> -DOGS_USE_PETSC=ON
• Build the source code with make

Create a structured mesh

In order to discretize the domain of a unit cube with boundaries execute bin/generateStructuredMesh -o cube_1x1x1_hex_axbxc.vtu -e hex --lx 1 --ly 1 --lz 1 --nx a --ny b --nz c where a, b, and c should be chosen according to the needs.

a	b	c	#cells in 10^6	compute cores	success
150	150	150	~ 3.38	20	yes
175	175	175	~ 5.36	20	yes
190	190	190	~ 6.86	20	yes
196	196	196	~ 7.59	20	no
216	216	216	~ 10.08	40	yes
236	236	236	~ 13.14	40	yes
292	292	292	~ 24.90	80	yes
368	368	368	~ 49.84	160	yes
422	422	422	~ 75.15	240	yes
465	465	465	~ 100.54	320

For the boundary conditions the boundary meshes left, right, top, bottom, front, and back were already constructed by the generateStructuredMesh tool. Alternatively, if the boundaries are simple (i.e. homogeneous), a .gml file is sufficient for the serial case, but for heterogeneous boundary conditions or for the parallelization boundary mesh files are needed.

There are two possibilities to create such files:

• If there is a .gml file, use constructMeshesFromGeometry tool which takes the mesh file and geometry and creates all the boundaries which are named in the .gml file with the required bulk_node_ids and bulk_element_ids mappings.
• If there is a boundary mesh (generated by Gmsh or Salome, or extracted in ParaView, for example) use the identifySubdomains tool to create or verify the needed bulk_node_ids and bulk_element_ids mappings.

Partition the mesh with partmesh

1. Convert VTU mesh into METIS input mesh
   bin/partmesh -i cube_1x1x1_hex_axbxc.vtu --ogs2metis
   => results in a file cube_1x1x1_hex_axbxc.mesh
2. partition the mesh and the corresponding boundaries.
   bin/partmesh -n number_of_partitions -m -i cube_1x1x1_hex_axbxc.vtu -- boundary_meshes*.vtu
   This will result in a bunch of .bin files.

Deciding the part to benchmark

• Usually, for solving elliptic problems (for instance, the steady state groundwater flow) the time for the linear solver is the main part of the simulation.
• The hydro-thermal process can be used for testing of the IO capabilities.
• In the mechanics process large parts of the computational resources are spent for the assembly.

ToDo

• script to create models and partitions
• submit script to cluster
• Metrics to take into account could be
  • weak scaling (problem size increases as the number of processors increases)
  • strong scaling (fixed problem size, use increasing number of processors => speedup and parallel efficiency)
  • fixed number of processors, increasing problem size