Official repository of:

• Class-Incremental Continual Learning into the eXtended DER-verse
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Semantic Residual Prompts for Continual Learning
• CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning

Mammoth is a framework for continual learning research. With more than 40 methods and 20 datasets, it includes the most complete list competitors and benchmarks for research purposes.

The core idea of Mammoth is that it is designed to be modular, easy to extend, and - most importantly - easy to debug. Ideally, all the code necessary to run the experiments is included in the repository, without needing to check out other repositories or install additional packages.

With Mammoth, nothing is set in stone. You can easily add new models, datasets, training strategies, or functionalities.

• Install with pip install -r requirements.txt. NOTE: Pytorch version >= 2.1.0 is required for scaled_dot_product_attention (see: Lightning-AI/litgpt#763). If you cannot support this requirement, uncomment the lines 136-139 under scaled_dot_product_attention in backbone/vit.py.
• Use ./utils/main.py to run experiments.
• New models can be added to the models/ folder.
• New datasets can be added to the datasets/ folder.

Mammoth currently supports more than 40 models, with new releases covering the main competitors in literature.

• Efficient Lifelong Learning with A-GEM (A-GEM, A-GEM-R - A-GEM with reservoir buffer): agem, agem_r
• Bias Correction (BiC): bic.
• Continual Contrastive Interpolation Consistency (CCIC) - Requires pip install kornia: ccic.
• CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning (CODA-Prompt) - Requires pip install timm==0.9.8: coda-prompt.
• Dark Experience Replay (DER): der.
• Dark Experience Replay++ (DER++): derpp.
• DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (DualPrompt) - Requires pip install timm==0.9.8: dualprompt.
• Experience Replay (ER): er.
• online Elastic Weight Consolidation (oEWC): ewc_on.
• Function Distance Regularization (FDR): fdr.
• Greedy Sampler and Dumb Learner (GDumb): gdumb.
• Gradient Episodic Memory (GEM) - Unavailable on windows: gem.
• Greedy gradient-based Sample Selection (GSS): gss.
• Hindsight Anchor Learning (HAL): hal.
• Incremental Classifier and Representation Learning (iCaRL): icarl.
• JointGCL: joint_gcl (only for General Continual).
• Learning to Prompt (L2P) - Requires pip install timm==0.9.8: l2p.
• LiDER (on DER++, iCaRL, GDumb, and ER-ACE): derpp_lider, icarl_lider, gdumb_lider, er_ace_lider.
• Learning a Unified Classifier Incrementally via Rebalancing (LUCIR): lucir.
• Learning without Forgetting (LwF): lwf.
• Meta-Experience Replay (MER): mer.
• Progressive Neural Networks (PNN): pnn.
• Regular Polytope Classifier (RPC): rpc.
• Synaptic Intelligence (SI): si.
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (SLCA) - Requires pip install timm==0.9.8: slca.
• Transfer without Forgetting (TwF): twf.
• eXtended-DER (X-DER): xder (full version), xder_ce (X-DER with CE), xder_rpc (X-DER with RPC).
• AttriCLIP: attriclip.
• Slow Learner with Classifier Alignment (SLCA): slca.
• Continual Generative training for Incremental prompt-Learning (CGIL): cgil
• Semantic Two-level Additive Residual Prompt (STAR-Prompt): starprompt. Also includes the first-stage only (first_stage_starprompt) and second-stage only (second_stage_starprompt) versions.

NOTE: Datasets are automatically downloaded in data/.

• This can be changed by changing the base_path function in utils/conf.py or using the --base_path argument.
• The data/ folder should not be tracked by git and is craeted automatically if missing.

Mammoth currently includes 21 datasets, covering toy classification problems (different versions of MNIST), standard domains (CIFAR, Imagenet-R, TinyImagenet, MIT-67), fine-grained classification domains (Cars-196, CUB-200), aerial domains (EuroSAT-RGB, Resisc45), medical domains (CropDisease, ISIC, ChestX).

• Sequential MNIST (Class-Il / Task-IL): seq-mnist.
• Permuted MNIST (Domain-IL): perm-mnist.
• Rotated MNIST (Domain-IL): rot-mnist.
• MNIST-360 (General Continual Learning): mnist-360.
• Sequential CIFAR-10 (Class-Il / Task-IL): seq-cifar10.
• Sequential CIFAR-10 resized 224x224 (ViT version) (Class-Il / Task-IL): seq-cifar10-224.
• Sequential CIFAR-10 resized 224x224 (ResNet50 version) (Class-Il / Task-IL): seq-cifar10-224-rs.
• Sequential Tiny ImageNet (Class-Il / Task-IL): seq-tinyimg.
• Sequential Tiny ImageNet resized 32x32 (Class-Il / Task-IL): seq-tinyimg-r.
• Sequential CIFAR-100 (Class-Il / Task-IL): seq-cifar100.
• Sequential CIFAR-100 resized 224x224 (ViT version) (Class-Il / Task-IL): seq-cifar100-224.
• Sequential CIFAR-100 resized 224x224 (ResNet50 version) (Class-Il / Task-IL): seq-cifar100-224-rs.
• Sequential CUB-200 (Class-Il / Task-IL): seq-cub200.
• Sequential ImageNet-R (Class-Il / Task-IL): seq-imagenet-r.
• Sequential Cars-196 (Class-Il / Task-IL): seq-cars196.
• Sequential RESISC45 (Class-Il / Task-IL): seq-resisc45.
• Sequential EuroSAT-RGB (Class-Il / Task-IL): seq-eurosat-rgb.
• Sequential ISIC (Class-Il / Task-IL): seq-isic.
• Sequential ChestX (Class-Il / Task-IL): seq-chestx.
• Sequential MIT-67 (Class-Il / Task-IL): seq-mit67.
• Sequential CropDisease (Class-Il / Task-IL): seq-cropdisease.

• ResNet18 on cifar100
• ResNet18 on TinyImagenet resized (seq-tinyimg-r)
• ResNet50 on ImageNet (pytorch version)
• ResNet18 on SVHN

@article{boschini2022class,
  title={Class-Incremental Continual Learning into the eXtended DER-verse},
  author={Boschini, Matteo and Bonicelli, Lorenzo and Buzzega, Pietro and Porrello, Angelo and Calderara, Simone},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2022},
  publisher={IEEE}
}

@inproceedings{buzzega2020dark,
 author = {Buzzega, Pietro and Boschini, Matteo and Porrello, Angelo and Abati, Davide and Calderara, Simone},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {15920--15930},
 publisher = {Curran Associates, Inc.},
 title = {Dark Experience for General Continual Learning: a Strong, Simple Baseline},
 volume = {33},
 year = {2020}
}

Expand to see the BibTex!

• CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning (BMVC 2024) paper

  @inproceedings{heng2022enhancing,
    title={CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning},
    author={Frascaroli, Emanuele and Panariello, Aniello and Buzzega, Pietro and Bonicelli, Lorenzo and Porrello, Angelo and Calderara, Simone},
    booktitle={35th British Machine Vision Conference},
    year={2024}
  }

• Semantic Residual Prompts for Continual Learning (ECCV 2024) paper

  @inproceedings{menabue2024semantic,
    title={Semantic Residual Prompts for Continual Learning},
    author={Menabue, Martin and Frascaroli, Emanuele and Boschini, Matteo and Sangineto, Enver and Bonicelli, Lorenzo and Porrello, Angelo and Calderara, Simone},
    booktitle={18th European Conference on Computer Vision},
    year={202},
    organization={Springer}
  }

• Mask and Compress: Efficient Skeleton-based Action Recognition in Continual Learning (ICPR 2024) paper code

  @inproceedings{mosconi2024mask,
    title={Mask and Compress: Efficient Skeleton-based Action Recognition in Continual Learning},
    author={Mosconi, Matteo and Sorokin, Andriy and Panariello, Aniello and Porrello, Angelo and Bonato, Jacopo and Cotogni, Marco and Sabetta, Luigi and Calderara, Simone and Cucchiara, Rita},
    booktitle={International Conference on Pattern Recognition},
    year={2024}
  }

• On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (NeurIPS 2022) paper code (Also available here)

  @article{bonicelli2022effectiveness,
    title={On the effectiveness of lipschitz-driven rehearsal in continual learning},
    author={Bonicelli, Lorenzo and Boschini, Matteo and Porrello, Angelo and Spampinato, Concetto and Calderara, Simone},
    journal={Advances in Neural Information Processing Systems},
    volume={35},
    pages={31886--31901},
    year={2022}
  }

• Continual semi-supervised learning through contrastive interpolation consistency (PRL 2022) paper code (Also available here)

  @article{boschini2022continual,
    title={Continual semi-supervised learning through contrastive interpolation consistency},
    author={Boschini, Matteo and Buzzega, Pietro and Bonicelli, Lorenzo and Porrello, Angelo and Calderara, Simone},
    journal={Pattern Recognition Letters},
    volume={162},
    pages={9--14},
    year={2022},
    publisher={Elsevier}
  }

• Transfer without Forgetting (ECCV 2022) paper code (Also available here)

  @inproceedings{boschini2022transfer,
    title={Transfer without forgetting},
    author={Boschini, Matteo and Bonicelli, Lorenzo and Porrello, Angelo and Bellitto, Giovanni and Pennisi, Matteo and Palazzo, Simone and Spampinato, Concetto and Calderara, Simone},
    booktitle={17th European Conference on Computer Vision},
    pages={692--709},
    year={2022},
    organization={Springer}
  }

• Effects of Auxiliary Knowledge on Continual Learning (ICPR 2022) paper

  @inproceedings{bellitto2022effects,
    title={Effects of auxiliary knowledge on continual learning},
    author={Bellitto, Giovanni and Pennisi, Matteo and Palazzo, Simone and Bonicelli, Lorenzo and Boschini, Matteo and Calderara, Simone},
    booktitle={26th International Conference on Pattern Recognition},
    pages={1357--1363},
    year={2022},
    organization={IEEE}
  }

• Class-Incremental Continual Learning into the eXtended DER-verse (TPAMI 2022) paper

  @article{boschini2022class,
    title={Class-Incremental Continual Learning into the eXtended DER-verse},
    author={Boschini, Matteo and Bonicelli, Lorenzo and Buzzega, Pietro and Porrello, Angelo and Calderara, Simone},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2022},
    publisher={IEEE}
  }

• Rethinking Experience Replay: a Bag of Tricks for Continual Learning (ICPR 2020) paper code

  @inproceedings{buzzega2021rethinking,
    title={Rethinking experience replay: a bag of tricks for continual learning},
    author={Buzzega, Pietro and Boschini, Matteo and Porrello, Angelo and Calderara, Simone},
    booktitle={25th International Conference on Pattern Recognition},
    pages={2180--2187},
    year={2021},
    organization={IEEE}
  }

• Dark Experience for General Continual Learning: a Strong, Simple Baseline (NeurIPS 2020) paper

  @inproceedings{buzzega2020dark,
   author = {Buzzega, Pietro and Boschini, Matteo and Porrello, Angelo and Abati, Davide and Calderara, Simone},
   booktitle = {Advances in Neural Information Processing Systems},
   editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
   pages = {15920--15930},
   publisher = {Curran Associates, Inc.},
   title = {Dark Experience for General Continual Learning: a Strong, Simple Baseline},
   volume = {33},
   year = {2020}
  }

Get in touch if we missed your awesome work!

• Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method (ICML 2024) [paper] [code]
• AGILE - Mitigating Interference in Incremental Learning through Attention-Guided Rehearsal (CoLLAs 2024) [paper] [code]
• Interactive Continual Learning (ICL) (CVPR 2024) [paper] [code]
• Prediction Error-based Classification for Class-Incremental Learning (ICLR 2024) [paper] [code]
• TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion (NeurIPS 2023) [paper] [code]
• Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation (NeurIPS 2023) [paper] [code]
• A Unified and General Framework for Continual Learning (ICLR 2024) [paper] [code]
• Decoupling Learning and Remembering: a Bilevel Memory Framework with Knowledge Projection for Task-Incremental Learning (CVPR 2023) [paper] [code]
• Regularizing Second-Order Influences for Continual Learning (CVPR 2023) [paper] [code]
• Sparse Coding in a Dual Memory System for Lifelong Learning (CVPR 2023) [paper] [code]
• A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm (CVPR 2023) [paper] [code]
• A Multi-Head Model for Continual Learning via Out-of-Distribution Replay (CVPR 2023) [paper] [code]
• Preserving Linear Separability in Continual Learning by Backward Feature Projection (CVPR 2023) [paper] [code]
• Complementary Calibration: Boosting General Continual Learning With Collaborative Distillation and Self-Supervision (TIP 2023) [paper] [code]
• Continual Learning by Modeling Intra-Class Variation (TMLR 2023) [paper] [code]
• ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysis (ICCV 2023) [paper] [code]
• CBA: Improving Online Continual Learning via Continual Bias Adaptor (ICCV 2023) [paper] [code]
• Neuro-Symbolic Continual Learning: Knowledge, Reasoning Shortcuts and Concept Rehearsal (ICML 2023) [paper] [code]
• Learnability and Algorithm for Continual Learning (ICML 2023) [paper] [code]
• Pretrained Language Model in Continual Learning: a Comparative Study (ICLR 2022) [paper] [code]
• Representational continuity for unsupervised continual learning (ICLR 2022) [paper] [code]
• Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (ICLR 2022) [paper] [code]
• Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (ICLR 2022) [paper] [code]
• New Insights on Reducing Abrupt Representation Change in Online Continual Learning (ICLR 2022) [paper] [code]
• Looking Back on Learned Experiences for Class/Task Incremental Learning (ICLR 2022) [paper] [code]
• Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach (CoLLAs 2022) [paper] [code]
• Consistency is the key to further Mitigating Catastrophic Forgetting in Continual Learning (CoLLAs 2022) [paper] [code]
• Self-supervised models are continual learners (CVPR 2022) [paper] [code]
• Learning from Students: Online Contrastive Distillation Network for General Continual Learning (IJCAI 2022) [paper] [code]

Pull requests welcome!

Please use autopep8 with parameters:

• --aggressive
• --max-line-length=200
• --ignore=E402

If you're interested in a version of this repo that only includes the original code for Dark Experience for General Continual Learning: a Strong, Simple Baseline or Class-Incremental Continual Learning into the eXtended DER-verse, please use the following tags:

• neurips2020 for DER (NeurIPS 2020).
• tpami2023 for X-DER (TPAMI 2022).