Slow or too much data?

[ofionnad]

Hi @kleok,

I am running your soil moisture code on a ISCE TopsStack. The soil inversion is taking a very long time however (after 1 day it is at 35% and 7-8 day completion).
The tops stack is about 750 GB in size, and my AOI is roughly 4-5 times larger than your example image of the ford dry lake, over a period of about 14 months of Sentinel-1 data.
(I previously ran it on a small 2km x 2km sample and it worked)

I am running it on a medium sized cluster (64 cores, 128 GB RAM).
It seems to be using all the cores, but they are only at 30-40% capacity.
It is only using about 1/3 of the available RAM.

Do you know what the bottleneck in the code would be? Read/write speeds?
If it is not # cores, RAM, or IO speeds, then it is possibly the code somewhere?
It would be great if I could achieve some sort of speedup. If you could point me in the right direction that would be great.

[kleok]

Hello @ofionnad,
The inversion part is indeed taking a very long time. I think that the main bottleneck in the code is this function that inverts soil moisture from interferometric observations.

INSAR4SM/insar4sm/inversion.py

Line 24 in 9caa75d

def inversion(SM0:np.array,

Based on my understanding and some experimentation, the computational time increases a lot the more observations (number of images) you provide. This is because the inversion scheme uses combinations of the provided images. So the more images you provide the bigger the number of combinations it becomes.
I suggest you to use to segment temporally our images from one dry season to the next dry one (e.g. a year span). Hopefully this will be helpful for speedup reasons without significant accuracy losses.
I hope my suggestion will help you 😃