Statistics > Machine Learning
[Submitted on 16 Jun 2020 (this version), latest version 5 Nov 2021 (v6)]
Title:Curvature is Key: Sub-Sampled Loss Surfaces and the Implications for Large Batch Training
View PDFAbstract:We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We show that the magnitude of the extremal values of the batch Hessian are larger than those of the empirical Hessian. Our framework yields an analytical expression for the maximal SGD learning rate as a function of batch size, informing practical optimisation schemes. We use this framework to demonstrate that accepted and empirically-proven schemes for adapting the learning rate emerge as special cases of our more general framework. For stochastic second order methods and adaptive methods, we derive that the minimal damping coefficient is proportional to the ratio of the learning rate to batch size. For adaptive methods, we show that for the typical setup of small learning rate and small damping, square root learning rate scalings with increasing batch-size should be employed. We validate our claims on the VGG/WideResNet architectures on the CIFAR-$100$ and ImageNet datasets.
Submission history
From: Diego Granziol [view email][v1] Tue, 16 Jun 2020 11:55:45 UTC (2,924 KB)
[v2] Fri, 6 Nov 2020 22:57:21 UTC (3,380 KB)
[v3] Sun, 31 Oct 2021 09:17:21 UTC (4,979 KB)
[v4] Wed, 3 Nov 2021 15:29:07 UTC (4,979 KB)
[v5] Thu, 4 Nov 2021 15:58:13 UTC (3,330 KB)
[v6] Fri, 5 Nov 2021 09:01:59 UTC (3,330 KB)
Current browse context:
stat.ML
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.