Implement Generation / Eval with deepspeed model engine

[sdtblck]

currently Generation / Eval are happening with the pytorch model, not the model engine. This is already causing memory problems and won't allow us to scale up - we'll need to implement this with the deepspeed model engine.

[StellaAthena]

Where in the code does this happen?

[srulikbd]

at the end of train_enwik8.py for example-the commented code.
I can try doing it but I'm not an expert in deepspeed yet.
should we use those examples?
https://github.com/microsoft/DeepSpeedExamples/blob/master/Megatron-LM/evaluate_gpt2.py
https://github.com/microsoft/DeepSpeedExamples/blob/master/Megatron-LM/generate_samples.py

[srulikbd]

but actually I see that it's already changed on train.py:
if params.get("validate_every") is not None: if is_main and i % params["validate_every"] == 0: model_engine.eval() with torch.no_grad(): val_data = next(val_loader).cuda() loss = model_engine(val_data) pbar.write(f'Validation Loss: {loss.item()}')
but not on:

if params.get("generate_every") is not None: if is_main and i % params["generate_every"] == 0: model.eval() val_data = next(val_loader).cuda() inp = random.choice(val_data)[:-1] prime = tokenizer.decode(inp) pbar.write(f"{prime} \n\n {'*' * 100}") sample = model.generate(inp.cuda(), params["generate_length"]) output_str = tokenizer.decode(sample) pbar.write(output_str)

[StellaAthena]

Huh, funny oversight. Yeah, push a patch to the generate function and we’ll close this issue.

[srulikbd]

actually it's not possible to just change "model" to "model_engine" for generation.
is it implemented here
https://github.com/microsoft/DeepSpeedExamples/blob/master/Megatron-LM/generate_samples.py
to generate using multiple GPUs?

[StellaAthena]

Superseded by codebase refactoring.