Add Profiler annotation logger

[upsj]

The logger adds timers for all important Ginkgo events based on the perfstubs library, VTune, nvTX and rocTX. To make it work, we also need to forward all logger events to the executor. This is meant to show how we can annotate a profile with Ginkgo-specific information.

An example profile from 10000 iterations in preconditioned-solver.cpp.
Not sure what's going on with spmv#3, but this may be related to inclusive vs. exclusive timing.

Below, you can see the same example running with NSight Systems and NVTX ranges.

[tcojean]

LGTM.

They also have this dynamic phase thing, I don't know if we can use this for solver iterations or what it does?
https://github.com/UO-OACISS/perfstubs/blob/master/perfstubs_api/timer.h#L136

I'm also not too happy to propagate everything to the executor as that defeats the purpose of the split system we have as that will make users rely much more on event masks to get only a subset of the events, instead of being responsibility based. But that is more of a preference than a requirement.

[upsj]

format-rebase!

[ginkgo-bot]

Formatting rebase introduced changes, see Artifacts here to review them

[MarcelKoch]

I think we should discuss, why this uses global loggers. As far as I can see, the logger itself does not to be a global object. Each hook just calls a function, so there is no need for global state. Only for Tau we need some global state, but that can be kept minimal.
So I think the global state is only used by the decision to propagate all loggers to the executor. Perhaps it's possible to achieve that without global state. Maybe we could specialize EnableLogging for Executor and add a counter of attached e.g. PropagatedLoggers to check if propagation is necessary.

[upsj]

I'm very open to other solutions to the global initialization problem - we have a similar issue with things like MPI_Init/Finalize, where users may or may not want us to take care of initialization. I do like your suggestion a lot, I didn't consider letting the Executor decide whether to receive propagated events. We could potentially also extend that to handle the split between host and device executor, making the workaround for the benchmarks unnecessary? But that would make the host executor the destination for all events, which is usually not that easy to access compared to the device executor, which is used everywhere.

[upsj]

On the global object question: I didn't realize until now, in case somebody were to run some kind of weird hybrid solve, it might actually make sense to have multiple different loggers with different properties all providing NVTX ranges. I'll add a bit of color to the nvtx example :)

[upsj]

Not using singletons for nvtx and roctx definitely makes sense: We can assign different colors to different executors that way (with a bit of corporate branding 😄 ): Yellow is GPU, gray is CPU

[upsj]

Random thought: A future feature that would work especially well for NVTX would be allowing users to give names to objects: profiler_set_name(linop.get(), "mass_matrix") would probably make the profile much more readable, and be very cheap to implement

[upsj]

ROCm profiling also seems to work :)

[ginkgo-bot]

Note: This PR changes the Ginkgo ABI:

Functions changes summary: 0 Removed, 291 Changed (57151 filtered out), 306 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

For details check the full ABI diff under Artifacts here

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
25 Code Smells

67.5% Coverage
3.1% Duplication