SparseAdam does not support with share_decoder_embeddings option

[wonkeelee]

Hi,

As far as I know is that to train transformer MT, set sparseadam to optimizer instead of adam through # 637
Therefore, When I tried to train transformer MT with -share_decoder_embeddings option, error occurred with message "SparseAdam does not support dense gradients, please consider Adam instead"

Is there any problem with changing the optimizer to adam according to the error message?

Thanks!

[vince62s]

Did you use multi gpu or single gpu ?

in any event multi gpu does not support sparse adam so use adam instead.

but we may need to fix sparse adam single gpu with share_decoder_emneddings
we may have missed somehting.

[wonkeelee]

@vince62s Thanks for your quick response. I worked with single GPU thanks!

[vince62s]

Can you please post your command line and error so that I can have a look ? (I think we may drop sparse adam in the future unless pytorch supports sparse for distributed functions)