[WIP] SamTov measure memory scaling

[SamTov]

Test this PR with 2 GB on

---

Fix #464 and fix the memory safety of MDSuite which is currently not robust enough for small memory machines (< 8 GB or so) on large data sets (>10 GB).

Memory scaling measurement methodology

It seems one can use the pytest-monitoring pytest plugin to measure the memory usage of specific tests. The outputs of these measurements are stored in an sqlite database that can then be queried for information about the specific test that was run. The workflow is outlined below:

1. Turn off memory management in a calculator
2. Add several experiments to a project, each with a different size of data in steps [1, 10, 50, 100] MBs. In the case of species wise computations different experiments are not needed however for flux data we cannot enforce computations over different groups and therefore, several experiments will be required.
3. Run the calculator for different databases and measure memory scaling w.r.t the dominant scaling axis
   • In the case of an atom-wise calculators this may be data range or number of atoms depending on the operation.
4. Fit a function to the relationship and try to evaluate the scaling function exactly.

This requires the person running the experiment to install pytest-monitoring locally. It can be added theoretically to the CI on GitHub and be tested every time but we will then have to make the SQL reading more robust.

Tasks

• Add general framework for memory testing using pytest monitor
• Add functionality for adding a hdf5 database directly into an MDSuite experiment
• Add functionality for turning off memory management in case of a test (all batch sizes to 1 and no minibatching)
• Add curve fitting to the test for identify correct relationship.
• Exclude the memory scaling test from the CI
• Adjust GitHub runners to use 4 GB memory machine to ensure that memory safety is achieved on a reasonably memory limitd device.

Reviewer notes

Any assistance with this is welcomed as it is not the only important PR on MDSuite at the moment, namely, #475 and this still need to be fixed before any release.

[PythonFZ]

I would have some questions:

Add functionality for adding a hdf5 database directly into an MDSuite experiment

What do you mean by that?

Exclude the memory scaling test from the CI

I did this for MLSuite with @pytest.mark.gap and then run pytest -m "not gap" and I think it makes sense to go with the same framework for MDSuite, but we could also name the files differently, idk.

Adjust GitHub runners to use 4 GB memory machine to ensure that memory safety is achieved on a reasonably memory limitd device.

The do have 7 GB of memory for linux / windows and I dont't think we can change that.

[PythonFZ]

Maybe we could expand the config to be config.memory.scaling_test = True instead of config.memory_scaling_test = True with additional dataclasses. This way it could be more structured.

[SamTov]

Yeah that's an interesting idea. Managing how configuration things are set in general is a nice thing to discuss as it can be quite involved. I think having data classes for different things like you mention here would be very nice.

[SamTov]

I would have some questions:

Add functionality for adding a hdf5 database directly into an MDSuite experiment

What do you mean by that?

So what I want to do, and what I have started in the test module, is to generate several hdf5 databases with data of different exact sizes, e.g 1MB, 10MB and so on. In this case, rather than generate a numpy array, save it to a readable file, and then read it with MDSuite, I want to make hdf5 database, make an experiment, and then add it as data to that experiment.

In practice, this would be the equivalent of saving your simulation data into the H5MD database format and then using it in MDSuite.

Exclude the memory scaling test from the CI

I did this for MLSuite with @pytest.mark.gap and then run pytest -m "not gap" and I think it makes sense to go with the same framework for MDSuite, but we could also name the files differently, idk.
So the only factor here is whether we want to have it run each time? I think the it is unnecessary to have the scaling stuff run every time but am open to differing opinions.

Adjust GitHub runners to use 4 GB memory machine to ensure that memory safety is achieved on a reasonably memory limitd device.

The do have 7 GB of memory for linux / windows and I dont't think we can change that.

I thought somewhere you can set a memory limit but maybe I am mistaken. In that case, we can set up local runners and just keep it as a test for say, releases that we perform locally.

[github-actions]

a262669

Memory Scaling

Raw data

Activate in workflow file