Loading in large file sets with multi-core processing and 'iris.load'

[dannymcculloch]

Hi everyone, I have a question regarding splitting iris.load across multiple processors, I had a look around but couldn't see anything that answers it. I'm fairly new to programming, so appreciate that the answer may be a simple one or I'm approaching it in the wrong way.

My question:
I'm using 1 processor and loading in a batch of files that takes ~48m, I'm loading in all the pg files across all outputs with:
dataset = iris.load(f'{path}/umglaa.pg*')

I'd like to reduce this time by using multiple processors (currently using 1, but have access to more). I've looked at parallelisation but I don't think that's entirely appropriate as I'm trying to create a singular large object, rather than multiple smaller processes (but please correct me if I've understood that wrong!). Can anyone recommend a way of increasing the processor count to use on a single command?

Bit more in-depth detail:
Context: I'm trying to load in all the pg files across multiple years of output twice (two test suites), currently I've run 1032 days (with more currently running which will also need to be loaded in) with each batch of 1032 days taking ~48 load time.
Aim: Distribute the load across multiple processors so that loading in the same batch takes ~24 minutes (in this hypothetical, it would be run on 2 processors instead of 1). The output would still be a single object though, callable later in the code (i.e. object=iris.load(pg) uses multiple processors so is faster, but object is still a singular object that is iterable)

Happy to provide more context if needed!
Thank you in advance!

[trexfeathers]

Hi @dannymcculloch, are you certain this isn't already happening? Iris is built on top of the Dask package, which in most (🤞) cases takes care of multi-processing automatically. I've just tested on some UM files, merging them together in Iris, and all my processors are engaged while this is happening.

Given this use of Dask, I'd advise extreme caution if you're thinking of adding your own layer of parallelism as they can interact unpredictably (with both layers attempting to engage all processors, giving you n-processes squared!).

If you need your script(s) to run faster, you should therefore hopefully be able to just throw more processors at the problem without any further work.

There are also opportunities for optimisation with Python, depending on your specific operations:

• Try to keep your data 'lazy' - Real and Lazy Data
• Note that UM data is stored as a series of 2D planes (lat-lon), repeated over other dimensions. This is how Dask will chunk its operations by default, so performance will be optimal if you can arrange your slicing and other operations to run within single 2D chunks - where possible.

[dannymcculloch]

Hi @trexfeathers, thanks for the response! So you're right that it's using all the processors, it's just that I got my terminology wrong I'm afraid (apologies). What I meant to ask was regarding multiple cores rather than processors, but all the processors within that core. So to correct my question:
I'd like to run iris.load across multiple cores to load in a single object (happy to leave Dask to handle the processor allocation though).

[trexfeathers]

I see. It's not a subject I know about I'm afraid. Iris' use of Dask will be no help either according to this comment.

This page recommends looking into Python multi-threading for such a case. Whatever you do I repeat my warning that you'll need to be careful to make sure your solution doesn't interact badly with Dask and lead to more threads than there are available processors/cores.

Sorry I can't be of more help. I've never looked at threading within cores since we generally find that there are greater opportunities to optimise the specific code being run to work more optimally with laziness and chunking.

[pp-mo]

Archiving "answered" Q+As

[dennissergeev]

Sorry @pp-mo, but would you be able to post a link to the relevant Q&A here please?