[help] targets + crew.cluster get stuck towards the end of the pipeline, but get unstuck if I restart #1276
-
Help
Description[premise] This issue has been recurring for a long time with different workflows and different software versions. I am using the most recent targets+friends ecosystem. I have a workflow with thousands of targets (targets file below), and after I leave it for few days it gets stuck as below ![]() It stays stuck as such for 20+ hours The HPC has still running jobs (apparently), so I would assume each target just takes a long time. ![]() However, If I stop the pipeline and relaunch it, I see that this is not the case! Targets finish in minutes and seconds, and the pipeline completes ![]() Perhaps setting It is not easy to to repex because this behaviour probably depends on the fact that this is a huge workflow. in fact, it only happens when the workflow is toward the end. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Beta Was this translation helpful? Give feedback.
tasks_max = 1
might help if your tasks are somehow accumulating objects in the global environments of the workers. So would worker-level storage management and memory optimization intargets
: https://books.ropensci.org/targets/performance.html