Request for Knowledge Distillation example

[mountains-high]

Hello, Thank you for the great work.

I'd like to ask for your assistance in implementing some of the experiments from https://arxiv.org/abs/1912.08795

For CIFAR-10 KD, ResNet-34, VGG-11 as a teacher network.
ResNet-18, VGG-11 as a student network.
Table 1 and Figure 4, could you show one example, please?

Thank you very much for your time and consideration.

[yoshitomo-matsubara]

Hi @mountains-high ,

Just a disclaimer, I am not very familiar with this paper, and probably you are more familiar with the paper than I am .
At first glance, it looks like their proposed approach has two steps: generating synthesized images, and knowledge distillation.
I take a quick look at the repository, and the implementation looks very complicated.

e.g.,
According to Fig. 1, the knowledge distillation looks a standard method (like this), but their implementation doesn't look like that.

To help you reimplement their approach with torchdistill, I need to understand what modules in models are trained and how (input-output, loss functions, etc).
If you could clarify these, I can give you a more concrete advice.

P.S.
Personally, I would not recommend CIFAR-10 experiments using ResNet-18 & -34 or VGG because those are designed and well benchmarked for ImageNet datasets. For more fair evaluations, I'd suggest models designed for CIFAR datasets like these if you want to use CIFAR datasets.

[mountains-high]

Hello there,
Thank you for your support. 
KD will be standard as you mentioned here (According to Fig. 1, the knowledge distillation looks a standard method (like this))

[yoshitomo-matsubara]

Hi @mountains-high ,

KD will be standard as you mentioned here

Are you sure that it is just a standard KD?
At least for CIFAR-10, their implementation doesn't look like that as mentioned above.

If you are not sure, I'd suggest contacting the authors at their repo (by creating a issue) or by email.

[mountains-high]

Good day, Dr. @yoshitomo-matsubara

You were right about loss function. Authors implemented different KD losses as that was their contribution.
However, is it possible to do it with just a standard KD?

Thank you

[yoshitomo-matsubara]

Hi @mountains-high

I am quite not sure what you mean by "possible to do it with just a standard KD".
If you want to use a standard KD with your synthesized dataset, you can do so by registering a new PyTorch dataset class and defining the dataset in a yaml file.

For instance, you can just edit the dataset entry in this sample config and run the experiment on CIFAR-10 with wide_resnet40_1 and wide_resnet40_4 as student and teacher models.