Last batch size error with self.target

[mohitsharma29]

Hey

Thank You for putting up an easy to use implementation of ELR on Readme.
I tried using it for my project. I had to make a few changes for that:

1. Removed the index from forward() function, to make this loss work for batches.

To make this work for batches, num_examp can be set to the batch size.
When your dataset size is not exactly divisible by the batch size, the last batch size will be less than num_examp and hence you should get an error while updating:
self.target = self.beta * self.target + (1-self.beta) * ((y_pred_)/(y_pred_).sum(dim=1,keepdim=True))

Now here is a very weird behavior: Batch size 32,64 didn't give any error but other batch size did, even though with batch size 32,64 the last batch is smaller than 32 (its 1).
A simple fix for that would be to instantiate the loss function during the training loop and with the current batch size.

What do you think?

[shengliu66]

Hey!

Thanks for your attention to our work! without the index, I am not sure how should we track the instances across different epochs. In order to make it work for batch, we need to make sure that the order of the examples is consistent within the batch for different epochs.

What does self.target you have mentioned refer to? the target t for each batch? then how should we deal with targets for other batches which should be different from the target for the current batch as well as targets for the batches in the next epoch?

[mohitsharma29]

Hey

Thank you for the prompt reply. I seem to have completely missed the point of index. Let me make that small fix in the dataloader and revert back.

[mohitsharma29]

Hey

I have fixed the training loop. and now everything is fine, any recommendations on tuning these above methods?

I used the default params. I see that my validation loss first increases then decreases (earlier the performance always saturated to some validation accuracy). I am right now added 40% synthetic noise to my dataset.

I am reading your paper as we speak, but some additional insights are always better.

[shengliu66]

Does your validation set also noisily labeled? The method is usually not very sensitive to beta, but the lambda often needs to be tuned.

[mohitsharma29]

Not really, validation is clean, but I will tune lambda some more. Thanks for the advice and a very usable code. Great work.