[FEATURE]: Add DMS metric

[JarvisGexiaoXu]

Hi Nathan,

Just curious would it be feasible to add MNNIST to the data options and add DMS (from Beta-VAE paper https://openreview.net/forum?id=Sy2fzU9gl) to metrics.
Just started learning PyTorch Lightning, curious about scalability.

Best,
JRS

[nmichlo]

Hi there,

MNIST is not specifically a disentanglement dataset. I would be hesitant to add it to disent because of this, also because there are many existing dataset loaders for this dataset.

These is an example that uses MNIST, if you want to look at how to load typical datasets: https://github.com/nmichlo/disent/blob/fe8b66dd0de0f8eefe6bd62c3739405492618517/docs/examples/mnist_example.py

As for DMS, I can open a TODO, however, would you be willing to help contribute this metric?

[JarvisGexiaoXu]

Hi Nathan,

I am a newbie in this field, probably I can start by reading the TF implementation of DMS first, to see if I am capable of this task.

Best,
JRS

[IceClear]

@nmichlo Hi, Nathan. Thanks for the awesome work. May I ask how to evaluate the metrics in this repo on MINST? Since there is no ground-truth, it is not possible to set self.__factor_sizes. But I think these metrics do not rely on labels for calculation. Is it possible to calculate these metrics on other datasets? Looking forward to your reply and thanks for your contributions!

[nmichlo]

Hi there @IceClear, thank you for the kind words!

Unfortunately it is not possible to calculate these metrics directly on MNIST, the dataset itself is not intended for disentanglement. Disentanglement datasets require that the datapoints have a one to one correspondance with the set product of the underlying individual ground-truth factors. MNIST unfortunately has more datapoints than there are unique labels.

I guess one way around this could be to treat the MNIST classes as ground-truth factors, but then sample randomly from all possible observations that have that same class as a label. Metrics could still possibly be computed off of such a dataset, but results might not work as expected.

My recommendation would be to stick to the usual datasets for the disentanglement metrics.

[IceClear]

Hi there @IceClear, thank you for the kind words!

Unfortunately it is not possible to calculate these metrics directly on MNIST, the dataset itself is not intended for disentanglement. Disentanglement datasets require that the datapoints have a one to one correspondance with the set product of the underlying individual ground-truth factors. MNIST unfortunately has more datapoints than there are unique labels.

I guess one way around this could be to treat the MNIST classes as ground-truth factors, but then sample randomly from all possible observations that have that same class as a label. Metrics could still possibly be computed off of such a dataset, but results might not work as expected.

My recommendation would be to stick to the usual datasets for the disentanglement metrics.

Noted. Thanks for your reply!