[Broken] Generation with Sequential Model

[satpalsr]

Take up a config and set "pipe-parallel-size": 1
Run python deepy.py generate.py configs/70M-deduped.yml -i input_prompt.txt -o prompt_out.txt
This will bring sequential model in action.

Error: 'SequentialWrapper' object has no attribute 'clear_cache' in line

[Quentin-Anthony]

I don't have the bandwidth to handle this for now. Would appreciate if someone can take a look.

[curt-tigges]

Will take a look!

[FourWinds021]

is this error solved now? I encountered the same problem when I train and inference a 125M GPT2 model according to the guidelines.

[StellaAthena]

is this error solved now? I encountered the same problem when I train and inference a 125M GPT2 model according to the guidelines.

If you encountered the same problem, it’s safe to say that it’s not solved.

[DaoD]

The same problem here. Could you please help to check it?

[yizhilll]

Same here. Would the temporary comment the line out affect anything?

[Quentin-Anthony]

Sorry all. This happened due to a line that snuck into the neox 2.0 release. To clarify for all, there are three pipeline parallelism cases in gpt-neox:

(pipe_parallel_size == 0): In this case, the model is wrapped in a standard nn.Sequential module

gpt-neox/megatron/training.py

Line 416 in c64bacc

if not neox_args.is_pipe_parallel:

This is done to reduce memory overhead and latency. This case is rarely used, and is where the issue you're seeing above lies.

(pipe_parallel_size == 1): This is the most common case. The model is wrapped in a single GPT2ModelPipe module, which makes it easier for both DeepSpeed and us to handle. This should be the default case, which we just resolved in #866.

(pipe_parallel_size > 1): This is for large models that require multiple pipeline module stages to distribute the model. This case remains unchanged.

What should you do: If you trained a model and need those weights to stay in a sequential module, we're working on a fix and will share it shortly. If you don't need those weights and instead simply need to run inference on a public model, apply the patch in #866 and run again.

[StellaAthena]

@satpalsr @FourWinds021 @DaoD @yizhilll have your issues been resolved by the recent patches?

[satpalsr]

yes