Skip to content
This repository has been archived by the owner on Mar 17, 2021. It is now read-only.

Maintains moving averages of all trainable variables #243

Open
wyli opened this issue Oct 11, 2018 · 0 comments
Open

Maintains moving averages of all trainable variables #243

wyli opened this issue Oct 11, 2018 · 0 comments

Comments

@wyli
Copy link
Member

wyli commented Oct 11, 2018

For many models trained with stochastic gradient descent, it is often beneficial to maintain a moving average of each model variable as the model evolves over training iterations. At inference time, the moving average version of the model is used instead of the model precisely at the last training iteration. This often gives better inference performance and should be implemented in the application driver (This will likely break the current variable restoring functions).

--originally posted at https://cmiclab.cs.ucl.ac.uk/CMIC/NiftyNet/issues/141

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant