The .grad attribute of a Tensor that is not a leaf Tensor is being accessed

[floatingbigcat]

Description
When training with what ever config, get warnning as

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /autofs/nccs-svm1_home1/quentin/eleutherai/pytorch/build/aten/src/ATen/core/TensorBody.h:482.)

This indicate that some non-leaf tensor is being accessed.
there is no such warnning in pure gpt-neox, and it still occur when I set add_adapter=False, therefore related to image_prefix at least.

[kshitijkg]

The reason the warning shows up is because images are floating point and deepspeed sets requires_grad to True for input images: https://github.com/EleutherAI/DeeperSpeed/blob/main/deepspeed/runtime/pipe/engine.py#L764

We can safely ignore this :)

Though this might lead to high memory consumption, will link required deepspeed PR here to avoid excessive memory consumption.