-
Notifications
You must be signed in to change notification settings - Fork 301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inefficient Gini Coefficient calculation? #855
Comments
Thanks for raising this issue. I am just about to leave for a week of travel and will definitely be looking into this. At first glance the efficient approach is based on a ranking of the values, which assumes no ties. I originally ruled that out as we had to both handle ties (in an upstream application) and also needed to keep tabs on geographical positions of each observation. So the memory inefficient implementation is what we used. That said, once I have a little time after vacation, I think it might be possible to refactor things with an eye towards a more efficient approach (as you point to) that also addresses our upstream needs. |
Have a nice time during your travels! I'm curious to see how you solve it. |
Closeed with #862 |
When I run:
where
a
is any huge numpy array, e.g., with 300000 elements, it gets stuck, irreparably crashing my whole PC, trying to run this line of code (see here):This is not the case when I run it using my own code which does not make use of list comprehensions, and instead calculates Gini like this.
I wrote that just before I discovered you had a Gini function too, so I do not fully understand your code, although I am willing to help. The two functions produce very very similar output (indistinguishable before about 6 decimal places).
Anyway, I thought I would mention that it be worth considering using numpy to calculate the difference.
The text was updated successfully, but these errors were encountered: