add some missing timeouts in Distributed #34502
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JeffBezanson
added
domain:parallelism
|
When we run out of file descriptors, I wonder if it would be too clever to try doing a full gc and then trying again... |
|
That sounds pretty good to me actually; potentially many places might need to be modified to do the error check and retry though. We could start by trying that in the specific places in Distributed that are failing. |
vtjnash
reviewed
Jan 27, 2020
vtjnash
approved these changes
Jan 27, 2020
JeffBezanson
force-pushed
the
jb/distributedtimeouts
branch
from
d7c8dae
to
06d0b9f
Compare
tanmaykm
added a commit
to tanmaykm/julia
that referenced
this pull request
Mar 10, 2020
fix typo in code that deals in timing out worker setup (introduced in JuliaLang#34502)
KristofferC
pushed a commit
that referenced
this pull request
Apr 11, 2020
tanmaykm
added a commit
to tanmaykm/julia
that referenced
this pull request
May 13, 2020
also the additional async task for timeout introduced in JuliaLang#34502 will not be required, because this PR handles that already and also differentiates between timeout and error.
tanmaykm
added a commit
to tanmaykm/julia
that referenced
this pull request
May 13, 2020
also the additional async task for timeout introduced in JuliaLang#34502 will not be required, because this PR handles that already and also differentiates between timeout and error.
tanmaykm
added a commit
to tanmaykm/julia
that referenced
this pull request
Apr 14, 2021
also the additional async task for timeout introduced in JuliaLang#34502 will not be required, because this PR handles that already and also differentiates between timeout and error.
Keno
pushed a commit
that referenced
this pull request
Jun 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will hopefully fix some of the intermittent hangs in mac CI. @Keno believes we are running out of file descriptors there, which I agree is a likely cause. That was causing an exception at an unexpected point, leaving some processes waiting forever. This should turn it into a hard fail. Then we just need to stop leaking descriptors, or get more :)
fixes #34486 (by converting the hang into an exception)