-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE]: Add DMS metric #33
Comments
Hi there, MNIST is not specifically a disentanglement dataset. I would be hesitant to add it to disent because of this, also because there are many existing dataset loaders for this dataset. These is an example that uses MNIST, if you want to look at how to load typical datasets: https://github.com/nmichlo/disent/blob/fe8b66dd0de0f8eefe6bd62c3739405492618517/docs/examples/mnist_example.py As for DMS, I can open a TODO, however, would you be willing to help contribute this metric? |
Hi Nathan, I am a newbie in this field, probably I can start by reading the TF implementation of DMS first, to see if I am capable of this task. Best, |
@nmichlo Hi, Nathan. Thanks for the awesome work. May I ask how to evaluate the metrics in this repo on MINST? Since there is no ground-truth, it is not possible to set self.__factor_sizes. But I think these metrics do not rely on labels for calculation. Is it possible to calculate these metrics on other datasets? Looking forward to your reply and thanks for your contributions! |
Hi there @IceClear, thank you for the kind words! Unfortunately it is not possible to calculate these metrics directly on MNIST, the dataset itself is not intended for disentanglement. Disentanglement datasets require that the datapoints have a one to one correspondance with the set product of the underlying individual ground-truth factors. MNIST unfortunately has more datapoints than there are unique labels. I guess one way around this could be to treat the MNIST classes as ground-truth factors, but then sample randomly from all possible observations that have that same class as a label. Metrics could still possibly be computed off of such a dataset, but results might not work as expected. My recommendation would be to stick to the usual datasets for the disentanglement metrics. |
Noted. Thanks for your reply! |
Hi Nathan,
Just curious would it be feasible to add MNNIST to the data options and add DMS (from Beta-VAE paper https://openreview.net/forum?id=Sy2fzU9gl) to metrics.
Just started learning PyTorch Lightning, curious about scalability.
Best,
JRS
The text was updated successfully, but these errors were encountered: