-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Executors: snooze on a lower level? #1576
Comments
That would definitely be helpful in our use case, we already have the problem for LT executors started from within Digital Micrograph. I am happy to look at how to do this. |
Great to hear! I was thinking that this could be handled partially by the Context, which would spawn the background task/thread which checks for the timer expiration, and then asks the executor to snooze, if it is supported. The pipelined executor (edit: or Maybe we can discuss this further tomorrow. |
Setting the snooze timeout very low, for example by starting with `libertem-server --snooze-timeout 0.001`, exposes some issues which this commit fixes, namely that the executor snoozes before `get_fs_listing`/`detect` finish running. It's not recommended to set the snooze timeout so small, but this change should increase stability in these cases - also possibly useful in cases where a lot of time passes between getting the executor and when the actual function is called, for example when the system is put to sleep, or possibly in case of forward jumps in time (?) Refs LiberTEM#1576 - lower-level keep-alive would prevent this kind of issue completely.
This also has the component that the executor needs to be kept alive across all those "running code/jobs/..." operations, which is best done in the executor itself (or maybe a wrapper kind of type), while, as noted above, high-level decisions about the timeout and concrete point of time of snoozing might be handled better by high-level code like |
Setting the snooze timeout very low, for example by starting with `libertem-server --snooze-timeout 0.001`, exposes some issues which this commit fixes, namely that the executor snoozes before `get_fs_listing`/`detect` finish running. It's not recommended to set the snooze timeout so small, but this change should increase stability in these cases - also possibly useful in cases where a lot of time passes between getting the executor and when the actual function is called, for example when the system is put to sleep, or possibly in case of forward jumps in time (?) Refs #1576 - lower-level keep-alive would prevent this kind of issue completely.
Another follow-up of #1572:
With #1572,
libertem-server
can temporarily shut down the whole executor. This solves the issue of long-running processes eating memory and CPU time very well, but a similar issue exists for dangling notebooks. We might want to add the same feature on the executor level, at least for the default dask executor, such that notebooks also free up their resources after a while. Important for shared systems, and especially when a GPU is shared between multiple users.The text was updated successfully, but these errors were encountered: