Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support passing arguments to underlying filesystem #522

Closed
wants to merge 3 commits into from
Closed

Support passing arguments to underlying filesystem #522

wants to merge 3 commits into from

Conversation

deepyaman
Copy link
Member

Description

Motivating Slack conversation:

Q: Is fs_args not supported for the overarching PartitionedDataSet? I need to specify endpoint_url for my S3 interface, and I can do this on other datasets (e.g. pandas.ParquetDataSet) but not on PartitionedDataSet. Without it, I can’t discover partitions.

A: Currently you can put those into credentials as this is the only set of kwargs passed into the filesystem constructor. Would it be acceptable for your use case? I will create a ticket to add fs_args support. Thanks for raising this 👍

Development notes

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
  • Added tests to cover my changes

Notice

  • I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

  • I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

  • I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

  • I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

@deepyaman deepyaman self-assigned this Sep 22, 2020
Copy link
Contributor

@DmitriiDeriabinQB DmitriiDeriabinQB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 But would also be great to add some unit tests, just to make sure that fs_args reach fsspec in the form that we expect. Let me know if any help needed.

@deepyaman
Copy link
Member Author

This branch is somehow orphaned, so I can't continue work here; moving to #559

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants