This repository is the implementation of our submission to the Tiny papers section of ICLR 2023.
The project contains 3 files and one notebook:
-
data.py - contains helpers and dataloaders for MNIST, CIFAR10 and FashionMNIST.
-
FFLayer.py - a pytorch implementation of a Forward Forward layer.
-
FF.py - a pytorch module that combines multiple FFLayers.
-
Notebook.ipynb - the main and only notebook that allows training an FF model for all of the implemented datasets and allows tweaking negative data generation and all other parameters.
In this study we explore the behaviour of the Forward Forward Algorithm by Hinton with separated positive and negative phases. We find a relation between the type of negative data and the behaviour at large separations.
Even though the CIFAR-10 dataset is included in this repo, it was not included in the study as it experienced close to no learning when the two phases were separated.
There's a clear performance advantage when using masks as negative data for larger phases.
| Dataset | Negative data | Unseparated | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|---|---|
| MNIST | Wrong label | 97.7% | 89% | 88% | 81% | 63% | 35% | 11% | 10% | 9% |
| MNIST | Masks | 96% | 88% | 84% | 85% | 78% | 74% | 49% | 23% | 11% |
| Fashion-MNIST | Wrong label | 88.7% | 73% | 63% | 59% | 54% | 20% | 14% | 10% | 10% |
| Fashion-MNIST | Masks | 84% | 56% | 58% | 53% | 50% | 45% | 36% | 30% | 22% |
The phase size of the positive data ranges from 1 to 16, and the negative phase is fixed at 1. The awake phase's learning rate is scaled down to have the same effect as the negative phase in the training procedure. The empty lines in the table represent models that experienced no learning.
| Dataset | Negative data | Unseparated | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|---|---|
| MNIST | Wrong label | 97.7% | 89% | 10% | 9% | - | - |
| MNIST | Masks | 96% | 88% | 78% | 75% | 73% | 10% |
| Fashion-MNIST | Wrong label | 88.7% | 73% | 56% | 47% | 14% | 10% |
| Fashion-MNIST | Masks | 84% | 56% | 55% | 54% | 52% | 49% |