-
Notifications
You must be signed in to change notification settings - Fork 947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added tag "bn_stat" to mean and inv_std params #837
Conversation
What's the use case? I'm fine with it if there's a good use. Should cover the cuDNN batch normalization layer as well, though. And I'd prefer |
I'm using it because I have an actual batch algorithm, where I want the statistics to be only from the current batch, in case I want to use more data than fits on one GPU to make the statistics. Then I manually get and set the stats over multiple calls to a forward pass function over slices of the data....so I need to grab these parameters out from the network. OK happy to change to batch_norm_stat. Where is the cuDNN layer? |
Note that you can achieve the same by changing the alpha value. Use alpha=1/(1+n) for the nth slice (starting with n=0) to average the statistics over multiple slices. (This uses Welford's algorithm then.)
👍
https://github.com/Lasagne/Lasagne/blob/39bc1b/lasagne/layers/dnn.py#L599 We might also want to add a test ensuring that you get back only the batch normalization statistics when calling |
I looked in the |
Oh! I was not aware it was possible to set |
I tried writing a short little test method onto the test class for the batch norm layer. But apparently something is wrong. Can you please check it out? I'm not really familiar with the test setup. |
Ah, correct. How thoughtful.
I don't remember how I did the cuDNN implementation in Theano, but at least for the regular batch normalization layer,
You instantiated a layer as |
Of course! Very nice, thanks I will use this. |
Oops, not sure why I would have done that, thanks for catching. Incidentally, I noticed that when you apply batch normalization to a layer using e.g.:
Then calling |
Exactly, it's because the The changes look good now, could you please squash everything into a single commit? git reset --soft b9e8c6f
git commit --amend -m 'Add "batch_norm_stat" tag to batch normalization mean and inv_std parameters' |
Seems handy.