Call ray.put in ray.init() to speed up first object store access.

[robertnishihara]

One problem that users often run into is that the first time they do something with Ray it is very slow. The second time is much faster. This applies to the first usage of any worker and not just the driver. The reason is that the first time a driver or worker accesses the object store (e.g., by calling ray.put or by completing a task), it takes about half a second (presumably to memory map the a large file).

Note that instead of calling ray.put, I tried just calling plasma_client.put instead. However, that only sped things up by a factor of 2.

import ray

ray.init()

%time ray.get(ray.put(1))

Before this PR: The timed line takes 500+ milliseconds.
Using plasma_client.put instead of ray.put: The timed line takes 250+ milliseconds (I'm not 100% sure why).
After this PR: The timed line takes 700 microseconds.

Note that ray.init() gets slower in this PR.

It's important that this happens on workers as well as the driver.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16967/
Test FAILed.

[edoakes]

LGTM. The behavior with the plasma client is odd, though.

[robertnishihara]

Jenkins, retest this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16977/
Test FAILed.

[robertnishihara]

Jenkins, retest this please.

[ericl]

Should we also do

@ray.remote
def f():
    return 1

ray.get([f.remote() for _ in range(multiprocessing.cpu_count())])

to warm up the workers? That might be overkill but otherwise we spend a lot of time waiting for workers to start as well.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16983/
Test FAILed.

[ericl]

Re: plasma put, maybe that's because ray put also does a get afterwards to pin the object.

[robertnishihara]

@ericl good question, the ray.put introduced in this PR should happen on all of the workers as well (though I should verify that before merging). But there might be some other things that need to be warmed up (e.g., the raylet to object store connection).

[robertnishihara]

Let's hold off on merging, some initial commands are still taking longer than expected, and I need to investigate that.

EDIT: The issue was just that I was testing on the wrong branch.

[robertnishihara]

@ericl I confirmed that this PR also warms up the workers.