-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Last batch size error with self.target #4
Comments
Hey! Thanks for your attention to our work! without the index, I am not sure how should we track the instances across different epochs. In order to make it work for batch, we need to make sure that the order of the examples is consistent within the batch for different epochs. What does |
Hey Thank you for the prompt reply. I seem to have completely missed the point of index. Let me make that small fix in the dataloader and revert back. |
Hey I have fixed the training loop. and now everything is fine, any recommendations on tuning these above methods? I used the default params. I see that my validation loss first increases then decreases (earlier the performance always saturated to some validation accuracy). I am right now added 40% synthetic noise to my dataset. I am reading your paper as we speak, but some additional insights are always better. |
Does your validation set also noisily labeled? The method is usually not very sensitive to beta, but the lambda often needs to be tuned. |
Not really, validation is clean, but I will tune lambda some more. Thanks for the advice and a very usable code. Great work. |
Hey
Thank You for putting up an easy to use implementation of ELR on Readme.
I tried using it for my project. I had to make a few changes for that:
To make this work for batches, num_examp can be set to the batch size.
When your dataset size is not exactly divisible by the batch size, the last batch size will be less than num_examp and hence you should get an error while updating:
self.target = self.beta * self.target + (1-self.beta) * ((y_pred_)/(y_pred_).sum(dim=1,keepdim=True))
Now here is a very weird behavior: Batch size 32,64 didn't give any error but other batch size did, even though with batch size 32,64 the last batch is smaller than 32 (its 1).
A simple fix for that would be to instantiate the loss function during the training loop and with the current batch size.
What do you think?
The text was updated successfully, but these errors were encountered: