BUG: fix isin with nans and large arrays

[Hanspagh]

Does a np.isnan if nan is given to isin and we have a large enough array to trigger the np.in1d path

• closes Inconsistent handling of nan-float64 in Series.isin() #22205
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[Hanspagh]

Seems tests are failing, but it seems unrelated to this?

[dsaxton]

Seems tests are failing, but it seems unrelated to this?

Yes, those are unrelated

[dsaxton]

Thanks @Hanspagh, can you add a release note for 1.1.3?

[Hanspagh]

Changes as requested, let mere know if there is anything else needed

[jreback]

pls also merge master, ping when green.

[Hanspagh]

Updated as requested

[jreback]

minor comments, ping on green.

[Hanspagh]

Fixed

[Hanspagh]

Done

[simonjayhawkins]

Thanks @Hanspagh minor nit re consistency of issue number comments

[Hanspagh]

I can do a rebase if needed?

[simonjayhawkins]

I can do a rebase if needed?

git pull will normally get the changes from the commit suggestion locally.

to update the PR with the latest changes on master see https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#updating-your-pull-request

[jreback]

thanks @Hanspagh

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

$ git checkout 1.1.x
$ git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 aed64e85eb17edb0e55013868b1aa4e44e977a36

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #36266: BUG: fix isin with nans and large arrays'

4. Push to a named branch :

git push YOURFORK 1.1.x:auto-backport-of-pr-36266-on-1.1.x

5. Create a PR against branch 1.1.x, I would have named this PR:

"Backport PR #36266 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.

[simonjayhawkins]

#36385 (comment) could have been responsible for not being able to auto backport

[realead]

Sorry a little bit too late. The whole point of doing this for len(comps) > 1_000_000, was that numpy was deemed to be faster (which is probably no loner the case btw, see #22205 (comment)), adding any, isnan, logical_or on top (with all the cache misses and temporary objects) will make this branch much slower. So probably it is best just to drop the whole branch and always keep f = htable.ismember_object (unless it is is_integer_dtype of cause).

[jreback]

can u run the asvs and check here?

[realead]

@jreback I have opened RP #36611 with my suggestion and some benchmarks, which show that numpy's in1d is only faster when here are very few unique values.