Julia seems to ignore process affinity settings on Linux

[oschulz]

Julia (tested with v1.0 and v1.3) currently seems to ignore (or escape from) process affinity settings on Linux. This is a problem in HPC computing contexts (but also in other scenarios), e.g. when trying to run one Julia process per NUMA domain.

Let's say we have a machine with 16 virtual cores (8 * 2):

$ numactl -s
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 
[...]

Now we use numactl to pin a process to CPUs 1 and 2, that process being a numactl -s to check that the limitation is actually in effect:

$ numactl --physcpubind=1,2 -- numactl -s
policy: default
preferred node: current
physcpubind: 1 2 
[...]

Now with numactl running numactl -s via julia:

$ numactl --physcpubind=1,2 -- julia -E 'run(`numactl -s`)'
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 
[...]

Within Julia, the pinning to CPUs 1 and 2 is gone. I see the same when running multi-threaded Julia code (Julia multi-threading or BLAS via Julia): All CPUs are used, instead of just the selected ones.

[Keno]

cc @kpamnany

[JeffBezanson]

We do call sched_setaffinity during init. But, it looks like that code was removed by #9639, then somehow got added back. It might have been a merge artifact when lots of threading work was happening on branches. So, we should just delete it again.

[oschulz]

Thanks, @JeffBezanson !

[mloubout]

This issue is still happenning

$ numactl --physcpubind=1,2 -- julia -E 'run(`numactl -s`)'
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 
cpubind: 0 
nodebind: 0 
membind: 0 
Process(`numactl -s`, ProcessExited(0))

[dominikkiese]

Same for me, on my hpc system, I find that julia schedules tasks to threads which should be permitted by the cpubind

[oschulz]

I usually use numactl, but since Julia v1.4, it has respected the given affinity.

[oschulz]

I get

$ julia -E 'run(`numactl -s`)'
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 
cpubind: 0 
nodebind: 0 
membind: 0 
Process(`numactl -s`, ProcessExited(0))

$ numactl --physcpubind=1,2 -- julia -E 'run(`numactl -s`)'
policy: default
preferred node: current
physcpubind: 1 2 
cpubind: 0 
nodebind: 0 
membind: 0