What's an example where two different tensors would have different values of width_mult?

[davisyoshida]

It seems to me that for the learning rate scaling etc. to work out properly, all infinite dimensions should be scaling linearly with "n", so for each d_i you have something like d_i = k_i * n. The width_mult value seems to be calculated by dividing this quantity for two different values of n, so shouldn't that give something like n_model / n_base for every tensor, regardless of its initial size?

[thegregyang]

Hi @davisyoshida, an example would be if you double the d_model of a Transformer but quadruple the d_ffn (where the MLP dimensions are d_model -> d_ffn -> d_model). Because we calculate width_mult using the fan-in dimension, this would cause the first nn.Linear.weight in the MLP layer to have its width_mult=2 but the 2nd nn.Linear.weight would have its width_mult=4. Nevertheless, as we demonstrate in our paper, we should expect hyperparameters to transfer even in this case.

[davisyoshida]

Got it, thanks!