gpt3small is broken

[StellaAthena]

gpt3small seems to have been left behind in some of our updates, and neither scripts/train_gpt3small.sh nor scripts/train_gpt3small_pipeline.sh run.

[srulikbd]

scripts/train_gpt3small.sh is running at my colab (I only dropped the train batch size because of OOM)..

[StellaAthena]

scripts/train_gpt3small.sh is running at my colab (I only dropped the train batch size because of OOM)..

Interesting. Can you provide a link to the file?

[srulikbd]

https://colab.research.google.com/drive/1IThG90kOdndybKuScNEZ1QG9nwnuj2l1?usp=sharing
there is some problem with the pipeline code. I'm trying fixing it. something with
num_stages (2) must divide distributed world size (1)

[StellaAthena]

Ah yes, you cannot use parallel pipeline on a single GPU unit. The whole point of parallel pipeline is to divide the pipeline between GPUs, which you cannot do. You can manually set num_stages to 1, or you can use the non-pipeline code.

…

On Wed, Jan 20, 2021 at 11:15 PM srulikbd ***@***.***> wrote: https://colab.research.google.com/drive/1IThG90kOdndybKuScNEZ1QG9nwnuj2l1?usp=sharing there is some problem with the pipeline code. I'm trying fixing it. something with num_stages (2) must divide distributed world size (1) — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZJVMBDQ76DLXFSFZZAWVDS26S75ANCNFSM4WHTG4SA> .

[srulikbd]

ok, what I suspected. thanks!
how can I help right now?
I saw that we need kubernetes skills, so I'm learning it.
anything else for a newcomer?

[StellaAthena]

Have you checked out the [Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)? That is extremely important to both this project and other ones, and is more accessible to people who can write code but are newcomers to AI research. Each task that needs completing has an associated Issue.

…

On Wed, Jan 20, 2021 at 11:53 PM srulikbd ***@***.***> wrote: ok, what I suspected. thanks! how can I help right now? I saw that we need kubernetes skills, so I'm learning it. anything else for a newcomer? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZJVMBE72SG77DM3M7KYG3S26XMFANCNFSM4WHTG4SA> .