Base.runtests: add environment variables for overriding the automatic networking detection

[DilumAluthge]

Closes #43211

Description

This pull request adds two environment variables for overriding the automatic networking detection that we perform before we run the test suite.

To force Julia to use multiple worker processes, even if Sockets.getipaddr() throws an exception, set the JULIA_TEST_USE_MULTIPLE_WORKERS environment variable to true.

To force Julia to use multiple worker processes AND to force Julia to run the tests that require network connectivity, even if Sockets.getipaddr() throws an exception, set the JULIA_TEST_NETWORKING_AVAILABLE environment variable to true.

Examples

Example 1

Suppose that:

1. Sockets.getipaddr() throws an exception when run in your build environment.
2. You want to use multiple worker processes to run the tests.
3. You do not want to run the tests that require network connectivity (Artifacts, Downloads, etc.).

To accomplish this, you would run the following commands:

export JULIA_TEST_USE_MULTIPLE_WORKERS=true
make testall

Example 2

Suppose that:

1. Sockets.getipaddr() throws an exception when run in your build environment.
2. You want to use multiple worker processes to run the tests.
3. You want to force Julia to run the tests that require network connectivity (Artifacts, Downloads, etc.).

To accomplish this, you would run the following commands:

export JULIA_TEST_NETWORKING_AVAILABLE=true
make testall

[vtjnash]

Can you add this to the help message from runtests?

[Apteryks]

Thanks for that, it is a neat workaround to the underlying problem (why should getipaddr fail when there's a loopback interface and a valid IP address set (127.0.0.1) ?). The same comment I made here: #43211 (comment). If getipaddr could be made to work normally in the container we wouldn't need JULIA_TEST_USE_MULTIPLE_WORKERS.

[DilumAluthge]

Can you add this to the help message from runtests?

I could, but that would mean making these environment variables part of the public API. I was thinking that because the use case here is very narrow, we could just keep it non-public API for now, which gives us the freedom to make breaking changes in the future.

[vtjnash]

I should have been more precise, I meant the help printed on line 84, which is not a stable API promise, but is the documentation of the current API:
https://github.com/JuliaLang/julia/pull/43217/files#diff-ed31af6984a988649fb6980f9a8bb124fbe2883fd886e0915d061004e48d33f0L84

If you want, you can also mention there that this is not suitable for use in scripts (it is "porcelain")

[zimoun]

What happens if JULIA_TEST_NETWORKING_AVAILABLE is set to true by the user but there is network?

The automatic network detection should be not bypassed. The only issue is being able to set n = min(Sys.CPU_THREADS, length(tests)) and currently it is because the test is only done with net_on which incorrectly reports the status of loopback; as detailed here #43205 (comment).

The correct fix is to fix getipaddr. But it is a deep change. Where the proposed PR #43211 is almost trivial and fix the immediate bug waiting a proper fix of getipaddr, IMHO.

This patch #43217 adds complexity and corner-cases without solving the real issue of getipaddr.

BTW, I do not understand why PR #43211 is closed without providing an explanation and why #43217 would be better (when it does not appear to be). Especially when explanations #43211 (comment) or #43211 (comment) discuss why #43217 does not look better than the trivial #43211.

[DilumAluthge]

If networking is truly not available, the user should not set JULIA_TEST_NETWORKING_AVAILABLE to true.

[DilumAluthge]

I don't completely follow your comments.

If I understand correctly, for your specific use case, Sockets.getipaddr() throws an exception, but you know that the loopback interface is actually available, and you want to use multiple worker processes to run the tests.

Is that a correct description of your specific use case?

If so, just follow the instructions below.

Example 1

Suppose that:

1. Sockets.getipaddr() throws an exception when run in your build environment.
2. You want to use multiple worker processes to run the tests.
3. You do not want to run the tests that require network connectivity (Artifacts, Downloads, etc.).

To accomplish this, you would run the following commands:

export JULIA_TEST_USE_MULTIPLE_WORKERS=true
make testall

[DilumAluthge]

#43211 was closed because it does not make sense to add logic for these very specific edge cases (e.g. Nix or GNU Guix) based on the presence or absence of the commonly-used JULIA_CPU_THREADS environment variable.

If we want to add logic for a rare edge case, we should add a dedicated environment variable for that edge case. That's exactly what this PR did.

[zimoun]

If I understand correctly, for your specific use case, Sockets.getipaddr() throws an exception, but you know that the loopback interface is actually available, and you want to use multiple worker processes to run the tests.

Is that a correct description of your specific use case?

Yes.

If so, just follow the instructions below.

I understand how to use the merged proposal. :-)

My point is: it is incorrect to bypass the network check. Because it is against the principle of least astonishment (POLA) for user interface.

Concretely, from my point of view, this test

    if !JULIA_TEST_NETWORKING_AVAILABLE
        try
            ipa = getipaddr()

is incorrect because POLA, whereas this test

if net_on || JULIA_TEST_USE_MULTIPLE_WORKERS
        n = min(Sys.CPU_THREADS, length(tests))

is correct because it fixes buggy getipaddr. Last, I am not convinced that another environment variable as JULIA_TEST_USE_MULTIPLE_WORKERS is necessary when it is the aim of JULIA_CPU_THREADS, but that's another story. ;-) And I trust you if you consider if it is better -- thanks for your explanations here #43217 (comment).

Anyway, talk does not cook the rice. ;-)

Thanks for the quick fix. Cheers!