Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataFrame] Implement loc, iloc #1612

Merged
merged 35 commits into from
Feb 27, 2018
Merged

Conversation

simon-mo
Copy link
Contributor

What do these changes do?

This PR implements basic loc and iloc for the dataframe.

Out of five allowed inputs, this PR implements the most commonly used three:

  • A single label, e.g. 5 or 'a', (note that 5 is interpreted as a label of the index, and never as an integer position along the index).
  • A list or array of labels, e.g. ['a', 'b', 'c'].
  • A slice object with labels, e.g. 'a':'f' (note that contrary to usual python slices, both the start and the stop are included!).
  • A boolean array.
  • A callable function with one argument (the calling Series, DataFrame or Panel) and that returns valid output for indexing (one of the above)

Related issue number

@simon-mo
Copy link
Contributor Author

simon-mo commented Feb 26, 2018

This PR depends on commit 634dba4 in PR #1599

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3976/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3988/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3989/
Test PASSed.

Copy link
Member

@devin-petersohn devin-petersohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of minor changes. Looking good!

df, idx_lst, col_idx): return df.iloc[idx_lst, col_idx]

retrieved_rows_remote = []
for partition, idx_to_lookup in lookup_dict.items():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to wrap this in list comprehension:

retrieved_rows_remote = [_deploy_func.remote(retrieve_func, self.df._df[partition], idx_to_lookup, col_list) for partition, idx_to_lookup in lookup_dict.items()]

from .dataframe import _deploy_func


def _row_tuples_to_dict(row_tuples):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be faster to do a self._index.groupby('partition')?

Copy link
Member

@devin-petersohn devin-petersohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thanks @simon-mo !

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4002/
Test PASSed.

@devin-petersohn
Copy link
Member

Passed on private-travis with unrelated failures. OK to merge. Thanks @simon-mo!

@devin-petersohn devin-petersohn merged commit 4ab16d7 into ray-project:master Feb 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants