This is the repository associated with the paper A path-norm toolkit for modern networks: consequences, promises and challenges by Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti and Rémi Gribonval, ICLR 2024.

• Setup with conda
• Reproduction of the results in [GBRG24]
• Functions of interest
• System information
• Acknowledgement and third-party notice
• License
• References

Follow these steps to set up the project environment using conda:

1. Clone the repository:

   git clone https://github.com/agonon/pathnorm_toolkit.git
   cd pathnorm_toolkit

2. Create a conda environment:

   conda create --name your_env_name

3. Activate the conda environment:

   conda activate your_env_name

4. Install pip within the environment:

   conda install pip

5. Install the project dependencies:

   pip install -e .

To deactivate and remove the environment when you're done:

conda deactivate
conda env remove --name your_env_name

We now describe the scripts to reproduce the different results of GBRG24. Whenever ImageNet is needed, precomputed results are available to save users from having to perform the computation from scratch. Options to perform the computations from scratch using ImageNet are described below and in each bash script.

• Time on my machine: 3 seconds.

If you decide to use the pre-computed value of B (max $L^\infty$ norm of the input images of ImageNet, normalized for inference), simply run:

bash scripts/0_compute_1st_part_bound_resnets.sh

• Time on my machine: 10 minutes on a single GPU V100 16GB.
• Memory requirements (can vary depending on your filesystem): 165GB (for ImageNet).

If you decide to recompute the value of B:

1. Open scripts/0_compute_1st_part_bound_resnets.sh in an editor.
2. Set data_dir to the ImageNet directory.
3. Adjust workers and batch_size variables.
4. Comment the line starting with --B.
5. Run bash scripts/0_compute_1st_part_bound_resnets.sh.

• Time on my machine: 40 seconds.
• Memory requirements (can vary depending on your filesystem): 0.7GB on your home directory used to download pre-trained ResNets available on Torch at the beginning of the script.

If you decide to use the pre-computed margins of all images of ImageNet for the different pre-trained ResNets of Torch, simply run:

bash scripts/1_compute_pretrained_pathnorm_margins_resnets.sh

• Time on my machine: 2 hours on a single GPU V100 16GB.
• Memory requirements (can vary depending on your filesystem): 165GB (for ImageNet), additional 0.7GB on your home directory used to download pre-trained ResNets available on Torch at the beginning of the script.

If you decide to recompute the margins of pre-trained ResNets on ImageNet:

1. Open scripts/1_compute_pretrained_pathnorm_margins_resnets.sh in an editor.
2. Set data_dir to the ImageNet directory.
3. Set saving_dir to where you want to save the results, ensuring at least 25MB free memory.
4. Adjust workers and batch_size variables.
5. Comment the line --margins-already-computed \.
6. Run bash scripts/1_compute_pretrained_pathnorm_margins_resnets.sh.

• Time on my machine: 1 minute 20 seconds.

If you decide to use the pre-computed metrics (path-norms, accuracy...) of a ResNet18 trained for 20 IMP iterations on ImageNet, simply run:

bash scripts/3_plot_imp.sh

• Time on my machine: 20 minutes per epoch (=~10mins/epoch of gradient update, and ~10mins/epoch to compute and save data margins after gradient update) on a single GPU V100 16GB, resulting in 30 hours per IMP iteration of 90 epochs and a total of 25 days for 20 IMP iterations of 90 epochs.
• Memory requirements (can vary depending on your filesystem): 165GB (for ImageNet), additional 0.2GB per IMP iteration for saving metrics and weights at best validation top-1 accuracy during training (4GB if you stay with the default 20 IMP iterations).

If you decide to retrain a ResNet18 for 20 iterations of IMP (Iterative Magnitude Pruning) on ImageNet :

1. Open scripts/2_train_imp.sh in an editor.
2. Set data_dir to the ImageNet directory.
3. Set saving_dir to where you want to save the results, ensuring at least 0.2GB free memory per IMP iteration.
4. Adjust workers and batch_size variables.
5. Run bash scripts/2_train_imp.sh.

To generate plots with your new results, open scripts/3_plot_imp.sh, set results_training_dir to match the saving_dir used in scripts/2_train_imp.sh, and run bash scripts/3_plot_imp.sh.

• Time on my machine: 3 seconds.

If you decide to use the pre-computed path-norms of a ResNet18 trained on 5 increasing subsets of ImageNet, simply run:

bash scripts/5_plot_increasing_dataset.sh

• Time on my machine: ~10/k minutes per epoch on a single GPU V100 16GB for 1/k of the full dataset. By default, the script trains for 90 epochs, on 1/2, 1/4, 1/8, 1/16 and 1/32 of the full dataset, each for 3 different seeds, resulting in a total of approximately 44 hours.
• Memory requirements (can vary depending on your filesystem): 165GB (for ImageNet), additional 0.33GB per seed and per training size for saving metrics and weights at best validation top-1 accuracy during training (5GB if you stay with the default 3 seeds and 5 training sizes).

If you decide to retrain a ResNet on increasing subsets of ImageNet:

1. Open scripts/4_train_increasing_dataset.sh in an editor.
2. Set data_dir to the ImageNet directory.
3. Set saving_dir to where you want to save the results, ensuring at least 0.33GB free memory per seed and per training size.
4. Adjust workers and batch_size variables.
5. Run bash scripts/4_train_increasing_dataset.sh.

To generate plots with your new results, open scripts/5_plot_increasing_dataset.sh, set results_training_dir to match the saving_dir used in scripts/4_train_increasing_dataset.sh, and run bash scripts/5_plot_increasing_dataset.sh.

Here are key functions within the project:

• Computing the Path-Norm of a model (see ok_models.json for valid ones):

  • Function: get_path_norm
  • Location: src/pathnorm/path_norm/compute_path_norm.py
  • Example of use: refer to scripts/compute_path_norm_models.py

• Computing the First Part of the Bound of Theorem 3.1 in GBRG24 for a ResNet:

  • Script: src/pathnorm/path_norm/compute_1st_part_bound_resnets.py
  • Example of use: refer to scripts/0_compute_1st_part_bound_resnets.sh

• Computing the Margins of a Model:

  • Function: get_all_margins
  • Location: src/pathnorm/path_norm/compute_margin.py
  • Example of use: refer to src/pathnorm/path_norm/pathnorm_and_margins_pretrained_resnets.py

• Train a ResNet with IMP:

  • Script: src/pathnorm/train_imagenet.py
  • Example of use: refer to scripts/2_train_imp.sh

These functions and scripts play a crucial role in various aspects of the project, from computing path-norms and model margins to training ResNets with IMP. Refer to the provided examples for correct usage and integration within your own workflows.

This code has been tested on a Linux system (February 2024) with the following hardware configuration:

• CPU: Intel(R) Xeon(R) Silver 4215R @ 3.20GHz
• GPU: NVIDIA A100 PCIe 40GB, NVIDIA V100 PCIe 16GB

conda 24.1
python 3.12
torch 2.2
torchvision 0.17
nvidia-cuda 12.1
numpy 1.26
pandas 2.2
matplotlib 3.8

The backbone of src/path_norm/train_imagenet.py (excluding stuff related to path-norms, margins, and iterative pruning) is an adaptation of the PyTorch example made in collaboration with Léon Zheng. Please review the NOTICE file for information about the license of the PyTorch example.

This project is licensed under the BSD 3-Clause License, see the LICENSE file for details.

[GBRG24]: Gonon, A., Brisebarre, N., Riccietti, E., & Gribonval, R. A path-norm toolkit for modern networks: consequences, promises and challenges. ICLR Spotlight 2024.