[WIP] Improve pyarrow-free remote-IO performance

[rjzamora]

Description

Improves fsspec-only behavior for multi-file and partial reads from remote storage. Host-memory usage will be suboptimal compared to NativeFile (for now), but performance will be comparable in most cases.

This PR also includes deprecations for pyarrow-based IO. However, #16132 should take priority for the deprecations (and this PR should be modified accordingly).

TODO

• Needs cleanup
• Must be revised to depend on Deprecate Arrow support in I/O #16132. Also, dask-cudf open_file_options deprecation should probably be pulled out and merged before both 16132 and this PR.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

Let's target 24.10. My view is, 24.08 we're deprecating but not removing NativeFile support. 24.10 we're removing, so we need to have some alternative in place, but it's OK if it's slower as long as we're working on a plan for improvement. Ideally by 24.12 we'd have something merged that at least restores performance parity.

[rjzamora]

Let's target 24.10. My view is, 24.08 we're deprecating but not removing NativeFile support.

Okay - targeting 24.10 for a real "behavior" change makes perfect sense to me. In 24.08, we are technically deprecating the user's ability to control whether or not NativeFile is used. I definitely like the idea of giving ourselves more time to actually implement the NativeFile replacement.

it's OK if it's slower as long as we're working on a plan for improvement. Ideally by 24.12 we'd have something merged that at least restores performance parity.

This PR is already comparable in most cases, and faster in some (e.g. many small files). Therefore, I'm confident we will have parity for 24.10 and hopeful that we will have something "better" for 24.12 :)

[rjzamora]

Closing as stale.