[Dataframes] Call ray.init() on ray.dataframe import

[kunalgosar]

Ensure ray is initialized when dataframes are imported.

The behavior here is interesting because the dataframe import is sent to all the workers. We need to ensure that the init is only run on the main thread.

[richardliaw]

you might want to try catch this just in case the user has already initialized ray in the code.

Also, what would be the ideal user experience in the cluster setting where ray has already started?

[kunalgosar]

Wrapped it in a try/except now; thanks!

The current experience is that it prints out the webui url. Is there some way to avoid this? By checking that ray has been initialized without a call to ray.init?

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4025/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4026/
Test PASSed.

[richardliaw]

I would actually advise against this. You might actually not want ray to take up all of the available CPU resources.

To answer your question, you would probably want to do something like ray.init(ray.services.get_node_ip() + ":6379") assuming the cluster is launched via the autoscaler.

Backtracking a little, what's the purpose of this PR? Perhaps there is a more elegant fix.

[devin-petersohn]

We are expecting our users to be traditional Pandas users, and one of the main drivers of our system is that you only have to change the import statement. I don't think most users will care about reducing their number of CPUs below the default. I don't see a problem with it for now, but I do agree that there are cases where users won't want access to all CPUs. We can add some docs for it later, but for now I think it's ok.

[richardliaw]

I see. Do you expect that the traditional user not to use multiple nodes then?

[devin-petersohn]

Right now we aren't supporting it, but eventually, yes. It's not a big deal for changes. We will put something more effective together later so customization is possible.

[richardliaw]

I see - thanks for clarifying.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4027/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4028/
Test PASSed.

[devin-petersohn]

Good catch! Thanks @kunalgosar

[robertnishihara]

We probably don't want the workers calling ray.init() right? Only the driver should do that.

When a data frame remote function is imported by a worker, the worker will try to do import ray.dataframe, which will cause the ray.init call to happen.

[devin-petersohn]

Right now we're only supporting dataframes locally until we can get a ray.put(other_machine).

[robertnishihara]

Right, I'm referring to worker "processes" as opposed to worker "machines".

[kunalgosar]

Is there some way to check that only the main thread on the main process runs the init function? It seems like the worker.py init check only looks at the name of the thread that is calling it.