"RuntimeError: CUDA out of memory" on lm-eval 0.3.0 through GPT-NeoX evaluate past a certain number of nodes #884
Labels
bug
Something isn't working.
duplicate
This issue or pull request already exists.
help wanted
Contributors and extra help welcome.
Hello,
I'm using the evaluate.py script from GPT-NeoX which loads the installed lm-eval 0.3.0 library. I'm using Megatron and DeeperSpeed on Summit, which has 6 V100s 16GB per node.
The issue is related to the number of nodes the evals run on, and the choice of task. For example,
hellaswag
runs fine with a 410M model evaluated on 2 nodes (but crashes with e.g. 8 nodes), butarc_easy truthfulqa_mc hellaswag hendrycksTest-*
will crash with the following trace:This model uses pp=mp=1; the same thing happens with other values, for example with a 7B model with higher mp and pp values. With
sciq
, it happens on 4 nodes but not 2. Pasting the args here:You can replicate this with the Pythia 410M model (at least on Summit). Another issue looks similar #849 even though they don't use GPT-NeoX I believe.
I've been talking to @Quentin-Anthony and @curt-tigges about this issue, so pinging them here in case they have additional information. The OoM issues do not happen because of loading the optimiser. The quantity of memory in excess scales with the number of nodes used. Every nodes signals an OoM issue. I'm guessing that the main worker of a pipelined model receives a constant quantity of info per pipelined model in the network during the accumulation of log likelihoods (i.e. with pp=1 on 2 nodes, it gets 2x , and 4x on 4 nodes etc), which is why it works until a certain number of nodes, varying with the eval task.
The text was updated successfully, but these errors were encountered: