-
Notifications
You must be signed in to change notification settings - Fork 51
Warning message: In quit... system call failed: Cannot allocate memory #322
Comments
Hi @ctlamb Yes, this means you will need a CPU with more memory. I suggest measuring the memory usage for each task so you have a benchmark of what Azure VM to use. Thanks |
Excellent, will do. In the meantime, I tried to use a machine with slightly more memory ("vmSize" = "Standard_E4_v3"),, but I am running into the following error after I run foreach (this error doesn't occur with "vmSize" = "Standard_DS12_v2")
|
What region are you in? It could be possible that Standard_E4_v3 is not available in your region. |
I'm in West US. The error throws in It looks like my tasks only use a max of 8GB of RAM, so the 28GB of ram I had in the "Standard_DS12_v2" should've been plenty. hmmmm. Not sure what's going on here Memory usage readout
|
Are you setting maxTasksPerNode greater than 1 in your cluster configuration? |
No it's =1
|
Is there a better/preferred package I could use to measure the memory usage? |
Now getting |
I don't have a preferred package for measuring memory usage. If you have a cluster configuration file and a reproducible sample, I will work on identifying the issue |
This is the same as issue #315. @ctlamb is your workflow using resource files uploaded to Azure storage? My workflow is and I haven't been able to determine whether the 7b error still occurs when not using resource files. I'd like to attempt to rule out whether resource files could be contributing in some way. |
Yes I am uploading and downloading data to Azure storage in my workflow. I do wonder if this was an internet issue? My internet speed was recently upgraded, and I haven't got the 7b error since..but thats only based on 5-10 different tries so far. Will update if anything changes |
Thanks for the extra information. My latest post at #315 documents the return of the dreaded 7b error. I considered your idea here as well. However, my university network is a gigabit connection and it's rock stable. My home internet is a 100Mb fire connection which is also super reliable (for the most part). I wonder if there's a limit to the number of connections Batch/HTTR can accept from a single IP address? I'm currently running 2 pools on my laptop (home network) and three on my uni workstation all day and they've been stable all day. If I try and run any more pools than this on either machine, the 7b error will return almost instantly. It's very strange... |
Are all of your workflows in interactive mode? (Waiting for the job to be done) Thanks, |
Mine is, yes. |
Any news on the status of this error? It's still happening to me with frustrating regularity. Thanks! |
Some of my nodes are failing with this error:
Warning message:
In quit(save = "yes", status = workerErrorStatus, runLast = FALSE) :
system call failed: Cannot allocate memory
Does this mean I need a CPU with more memory?
The text was updated successfully, but these errors were encountered: