[Data][Docs] Revise "Loading data"

[bveeramani]

Why are these changes needed?

The "Loading data" guide contains verbose examples and disorganized subsections. This PR abridges the guide and restructures the content.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[amogkam]

Can you add a PR description?

[raulchen]

just curious, does these directory exist in the CI env?

[bveeramani]

Not yet, but I'm planning on adding it in a follow-up PR. Thinking of download the s3:https://ray-example-data bucket to CI.

[raulchen]

IIUC, the following example will create a new connection for each ray.data.read_sql. This is an anti-pattern in practice. Do we want to update it to re-use one connection?

[bveeramani]

It's necessary to create a new connection because connections aren't thread or process safe. So, we can't share a connection across read tasks.

[raulchen]

Those read_sql calls are sequential. So multi-threading isn't a problem?

[bveeramani]

I'm not sure if I'm misunderstanding? The read_sql calls are sequential, but read_sql creates connections in read tasks. We create a connection in each read task so that we can read the database in parallel.

[ericl]

Edits look good, but we also have to somehow retain the performance section. The most common pain point from users is how to obtain good performance from Ray Data, so we have to be deliberate in addressing this need.

[amogkam]

For performance, we can just link to the appropriate section in the performance guide in the intro? https://docs.ray.io/en/master/data/performance-tips.html. And make sure that information is up to date.

Don't think we need to have it in both places

[ericl]

For performance, we can just link to the appropriate section in the performance guide in the intro? https://docs.ray.io/en/master/data/performance-tips.html. And make sure that information is up to date.

Currently, the performance page is a mis-mash of random / advanced tips. It certainly needs to be updated to focus on the basics for loading / transforming.

[bveeramani]

For performance, we can just link to the appropriate section in the performance guide in the intro? https://docs.ray.io/en/master/data/performance-tips.html. And make sure that information is up to date.

Currently, the performance page is a mis-mash of random / advanced tips. It certainly needs to be updated to focus on the basics for loading / transforming.

We're planning on revising the performance page in the near future. I'll leave the performance considerations in for now, and remove it once we've revised the performance page.

[amogkam]

This is great!!

Just last thing, per @ericl's comment let's add a "Creating synthetic datasets" section at the bottom that shows range, range_table, and range_tensor (just like what we have currently), and specifies this is useful for performance benchmarking.

[ericl]

As Amog's comment above suggested could we add range() and range_tensor() back as recommendations for creating large synthetic datasets for performance testing?

[bveeramani]

As Amog's comment above suggested could we add range() and range_tensor() back as recommendations for creating large synthetic datasets for performance testing?

@ericl yeah, just pushed the changes.