Scaling law often cause nan training loss when extend grids.

[plyu3]

I am trying to use the scaling law experiments and use extend grids as the grid number goes along [5, 10, 20, 50, 100]. The model trained well at the start when grid=5, but it gets nan value when I extend the grid size to 10, 20 after training grid=5.

[KindXiaoming]

does the dataset have any weird statistics?

[KindXiaoming]

I did observe nan sometimes, but there could be many reasons for it: data could have singularities, could be degenerating along some dimensions, LBFGS optimization can go wrong with large grids. I'm happy to provide more guided guess if you reopen the issue and provide more details.