Is PP = 1 faster than Sequential?

[StellaAthena]

As described in #573, there is a note in the code that if one sets PP = 1 the code will actually build a sequential model rather than a pipeline parallel model with one stage. It makes intuitive sense to me that a sequential model would be faster than a PP = 1 model because the pipelining has some overhead, but the authors of that issue report the opposite.

We should systematically test whether pipelining with one stage is faster than using a sequential model. If it isn’t, then we should consider changing it so that PP = 1 sequentializes the model. If it’s not faster, it may be worth keeping PP = 0 as a sequential model because there are times that the pipeline engine is difficult to use.

either way, we should clearly document the functionality and make the comments correct.

[sdtblck]

I tested this ages ago, there is not much noticeable difference, the overhead is quite small.

Also, slight mistake, you said

there is a note in the code that if one sets PP = 1 the code will actually build a sequential model rather than a pipeline parallel model with one stage.

it's the other way round. PP=0 == sequential, PP=1 == pipe parallel module

[sdtblck]

See this pr #269 for details. Going to close this, as I think the linked thread should answer your questions.