"source"
encoding for datasets opened from fsspec
objects
#8923
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When opening files from path-like objects (
str
,pathlib.Path
), the backend machinery (_dataset_from_backend_dataset
) sets the"source"
encoding. This is useful if we need the original path for additional processing, like writing to a similarly named file, or to extract additional metadata. This would be useful as well when usingfsspec
to open remote files.In this PR, I'm extracting the
path
attribute that mostfsspec
objects have to set that value. I've considered usingisinstance
checks instead of thegetattr
-with-default, but the list of potential classes is too big to be practical (at least 4 classes just withinfsspec
itself).If this sounds like a good idea, I'll update the documentation of the
"source"
encoding to mention this feature.whats-new.rst