You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As the last TP paper points, muP works (empirically) over depth too. Does the current repo support that? Basically, set_base_shapes zips a base model and a wider model, and as such can set the p.infshape for each param p. How can this be adapted to support depth?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello,
As the last TP paper points, muP works (empirically) over depth too. Does the current repo support that? Basically,
set_base_shapes
zips a base model and a wider model, and as such can set thep.infshape
for each paramp
. How can this be adapted to support depth?Thanks!
Beta Was this translation helpful? Give feedback.
All reactions