Issues with core usage when running group-level analyses on high-performance (HPC) system

[IanSOlivier]

Hi!

I've been trying to run group-level analyses/group statistics as part of an ENIGMA study on a high-performance computing (HPC) cluster. It looks like it runs well but then I have been getting repeated complaints from the HPC's IT support staff that HALFpipe group-level analyses are taking over more cores than reserved and thereby encroaching on other people's jobs on the system.

I was wondering if there is any advice for me that I can somehow stop HALFpipe from using more cores or limits its core usage. I used the following steps (as described in the HALFpipe GitHub page section(https://github.com/HALFpipe/HALFpipe#running-on-a-high-performance-computing-cluster).

1. Start "screen" from the head node
2. Request an interactive job and limit to 4 cores: sintx --ntasks=4
3. Run: singularity run --containall --bind /:/ext halfpipe-halfpipe-latest.sif --only-model-chunk and then select Run without modification in the user interface.
4. The job starts running
   However, the issue I get is described as:
   srvcnthpc102 olvian001: Using=7 *Reserved=4 srvcnthpc102

Or, if I don't specify number of nodes: srvcnthpc122 olvian001: Using=3 *Reserved=1 srvcnthpc122

Thanks in advance for the help!

[HippocampusGirl]

Hi @IanSOlivier,

HALFpipe does not handle node allocations by itself, it always depends on the settings you specify when requesting the allocation. I think it would be best to consult your cluster's documentation/work with the staff to solve the issue. When running group statistics, I think it should be sufficient to request just one node.

[IanSOlivier]

Hi @HippocampusGirl

Thank you so much for the response! I realised I had mistakenly written about the issue being node allocation, when the issue is actually that HALFpipe uses more cores in a node than specified (I request one node with 4 cores and it uses more). However, I suspect that the issue would still be the same as HALFpipe doesn't handle the number of cores allocated either, and I can only specify how many cores to use when running HALFpipe after requesting an interactive job/submitting a job in a submission script. I'll see if I can resolve with the staff. Thank you for the help!

[HippocampusGirl]

Hi @IanSOlivier,

You can control the number of cores used by adding the option --nipype-n-procs 4 (or any other number) to the end of the command line you use to start HALFpipe.

[IanSOlivier]

Ah wonderful! Thank you so much - that's perfect!