Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename classifier (for NRE) to critic #1103

Closed
wants to merge 11 commits into from
Closed

Rename classifier (for NRE) to critic #1103

wants to merge 11 commits into from

Conversation

bkmi
Copy link
Contributor

@bkmi bkmi commented Mar 22, 2024

What does this implement/fix? Explain your changes

The name classifier is inaccurate. Rather, a general name such as critic is more appropriate. The critic can be transformed into a classifier using the sigmoid (or other transformation). It can be made to estimate the likelihood-to-evidence ratio by simple evaluation. However, I think keeping the name of the network distinct from the RatioEstimator class is a good idea.

Does this close any currently open issues?

#1079

Any relevant code examples, logs, error output, etc?

Nope

Any other comments?

This cannot be applied until #1097 has been pulled!

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask. We're here to
help! This is simply a reminder of what we are going to look for before merging
your code.

  • I have read and understood the contribution
    guidelines
  • I agree with re-licensing my contribution from AGPLv3 to Apache-2.0.
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have reported how long the new tests run and potentially marked them
    with pytest.mark.slow.
  • New and existing unit tests pass locally with my changes
  • I performed linting and formatting as described in the contribution
    guidelines
  • I rebased on main (or there are no conflicts with main)

Copy link
Contributor

@janfb janfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! thanks a lot Ben 🙏
looks all good. I am just not sure about the renaming to "critic". Technically, I see your point. But I fear it will be less clear for users. It also breaks the API. But I am open to be convinced.

docs/docs/reference.md Outdated Show resolved Hide resolved
@@ -119,32 +92,32 @@ def build_linear_classifier(
check_embedding_net_device(embedding_net=embedding_net_y, datum=batch_y)

# Infer the output dimensionalities of the embedding_net by making a forward pass.
x_numel = embedding_net_x(batch_x[:1]).numel()
y_numel = embedding_net_y(batch_y[:1]).numel()
x_numel = embedding_net_x(batch_x[:1]).size(-1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the reason for this change? Was this a bug?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.numel() returns the number t.flatten().shape()[0], but this gives only the last dimension. I like thinking about dimensions when I can. Personally I think this makes more sense since we force the user to give (Batched) 1d inputs to these networks.

sbi/neural_nets/critic.py Show resolved Hide resolved
sbi/neural_nets/ratio_estimators.py Show resolved Hide resolved
sbi/neural_nets/ratio_estimators.py Show resolved Hide resolved
sbi/neural_nets/ratio_estimators.py Outdated Show resolved Hide resolved
sbi/neural_nets/ratio_estimators.py Outdated Show resolved Hide resolved
sbi/neural_nets/ratio_estimators.py Outdated Show resolved Hide resolved
@bkmi
Copy link
Contributor Author

bkmi commented Mar 25, 2024

I will follow up after Friday.

@bkmi
Copy link
Contributor Author

bkmi commented Apr 9, 2024

@janfb, I used critic since it was shorter than discriminator. Also critic is the name used in WGAN where the "discriminator" network does not act as a binary classifier. We are in that situation, so I chose the name critic. However, I don't have a strong opinion except not to call it a classifier. There's a slight bias against discriminator for me since discriminators in GANs are classifiers.

let me know how you like it and I'll change to to that so we can get this merged.

@bkmi bkmi requested a review from janfb April 9, 2024 14:31
Copy link
Contributor

@janfb janfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the update @bkmi !

I see your point of classifier vs discriminator vs critic. However, changing the name would result in a major API change. We would need to make sure it's back compatible, i.e., catch all classifier in arguments and (user facing) function names etc. and issue a deprecation warning.
I would be hesitant in doing all this unless we really need to :)
@michaeldeistler what do you think?

sbi/neural_nets/ratio_estimators.py Show resolved Hide resolved
@michaeldeistler
Copy link
Contributor

I would also prefer to not change this right now -- users will already get a bunch of warnings when the new version is released (thinning for MCMC, importing posterior_nn) so I would prefer to not change this.

@bkmi
Copy link
Contributor Author

bkmi commented Apr 19, 2024

okay, I'll follow your lead, but I would argue that maybe having one big change with many warnings will be simpler than adapting code once, then again for a later additional change.

@janfb
Copy link
Contributor

janfb commented Jul 2, 2024

Hi @bkmi to move on with this PR I suggest the following:

As we would prefer to not do the renaming to critic, we should drop those changes. The PR also contains a couple of other changes that we do want to keep though. However, it seems that those changes are contained in your other PR as (see #1097 ). Thus, could you please close this PR and double check that all not renaming related changes end up in #1097 ?

Thanks a lot! 🙏

@janfb janfb mentioned this pull request Jul 2, 2024
8 tasks
@bkmi bkmi closed this Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants