EPIC: Refactor pip package

[MalteEbner]

Current state:

Currently, we offer 3 levels of SSL parts:

• low-level: the separate building blocks such as a projection head, nn memory bank, backbone creation
  

  lightly/lightly/models/modules/heads.py

  Line 12 in 8dde4d6

  class ProjectionHead(nn.Module):

• mid-level: the model (backbone + heads + forward functions)
  

  lightly/lightly/models/moco.py

  Line 13 in 8dde4d6

  class MoCo(nn.Module, _MomentumEncoderMixin):

• high-level: the model, criterion, training loop put together as pytorch lightning module
  

  lightly/docs/source/getting_started/benchmarks/cifar10_benchmark.py

  Line 144 in 8dde4d6

  class MocoModel(BenchmarkModule):

Target state:

We remove the mid-level representations (delete that code). We provide more examples on the high-level: One example pytorch lightning module for every implemented paper

Steps towards it

Always merge into the development branch: https://github.com/lightly-ai/lightly/tree/master-refactor-SSL-models

First step:

• Refactoring: write new momentum encoder (make it more flexible) #494
• Refactoring: implement high-level example for one new model (e.g. SwAV) #492
• Refactoring: deprecation warnings for old models #498
• Update imagenette benchmark to use the new MoCo and BYOL models #505
• update benchmarks with the new models #508
• Refactor CLI (deprecate high-level models) #512

Second step:

• ensure compatibility with old pretrained models (checkpoints) #515
• Remove mid- to high-level code from CLI
• Update docs and tutorials after refactoring #380

Third step:

• delete mid-level models in lightly/lightly/models

[MalteEbner]

What is still unclear for me is which versions will belong to which steps. We should have clear plans when what will deprecate. In particular, we need published python versions where deprecations warning are raised, but the old models still work.

E.g:

• versions with 1.1.xxxx: Old models work without deprecations warnings. This is the state before this issue.
• versions with 1.2.xxxx: New models are working. Old models are still there and working, but raise deprecation warnings. This equals step 1 of this EPIC. We should use the branch master-refactor-SSL-models for it.
• versions with 1.3.xxxx The old models are removed. This equals step 2 of the epic. We should use a new branch for it.

[philippmwirth]

Requirements for 1.2.0, goal: mid-level models are removed

• Documentation and tutorials are up-to-date (no mid-level models are used)
  • one issue per tutorial
  • one issue for rest of docs
• Ensure CLI tool works (doesn't use mid-level models)
• Remove mid-level models
• Refactor loss function (make symnegcosine non-symmetrical)

[guarin]

The cli still uses SelfSupervisedEmbedding objects. We should decide whether we want to continue using it or also change to low level.