Performance benchmarks

[samjvsutcliffe]

Preface

I'm looking to use cb-geo/mpm as a foundation for my PhD, and had a performance issue come up.
I've built MPM for our HPC cluster (Hamilton 8), and found a performance 100x slower than reported in "Scalable and modular material point method for large-scale simulations" over the plate in hole benchmark.
I attempted to use the same parameters (No MPI, 16 threads).
The processor used in our cluster is a 2x AMD EPYC 7702 instead of the 2x Intel Xeon Skylake 6142 but the performance is reasonble.

Build setups

I followed the build guide in the Readme, with the exception of changing to a Release build (I checked it was using -O3).

Relevent modules loaded for the build:

• gcc/11.2
• mkl/2019.6
• boost/1.77.0
• eigen/3.3.9
• vtk/9.2.2
• HDF5 was built from source due to flag incompatibilities: version 1.13.4-1

Not built with:

• KaHIP
• Partio

A 100x performance regression is rather large, I'm thinking I've built something wrong but can't see where.

Profiling

Just doing some cursory profiling, I found the bulk of my time was spent in reasonble functions, and not some misconfigured library doing something weird.

Question

Does this performance seem weird to you, have I miss-built my setup and if so how can I build it properly!
Potentially this could be put down as an "issue" if there is a signifigant performance regression on AMD CPUs.

[kks32]

Are you compiling on Release mode? so it is not in debug mode? cmake -DCMAKE_BUILD_TYPE=Release .. ?
Could you try running with -p 1 flag, that is just one thread and see if it is a threading issue?

[samjvsutcliffe]

I've run the build, and found that it isn't using more than 1 core.
I've tested my gcc & openmp and that they work and can use multiple cores.

I added some debug output and found it is trying to use multiple threads, but they end up on one core.

I'm guessing openmp is getting configured somewhere in the build that doesn't line up with our OS/system (Rocky linux).

[kks32]

Which OpenMP version is this?

[samjvsutcliffe]

I'm using GCC version 11.2, which has support for OpenMP 4.5, and lots of OpenMP 5.

I'll try and get a more modern GCC compiler, but that might be hard.

[kks32]

Maybe the problem is small so having multiple MPI may help over having more OpenMP threads

[samjvsutcliffe]

I've hunted down the issue, its related to the MKL openmp implementation.
When building with gcc + mkl on the HPC architecture cb-geo ends up linking with both GCCs' libgomp & MKLs' iomp5 libraries.
When compiling with MKL disabled, I get better performance and code is actually running multithreaded (in a non MPI case).

Not sure if this is just related to my operating system/architecture but it seems there are ways to fix this as described in this stackoverflow page.
https://stackoverflow.com/questions/25986091/telling-gcc-to-not-link-libgomp-so-it-links-libiomp5-instead

I'm going to play around with cmake to try and fix this issue as it may affect other AMD architectures?

Overview: too many openmp libraries disable the multithreading