[serve][docs] Add dev workflow page #27746

architkulkarni · 2022-08-10T15:39:59Z

Why are these changes needed?

Adds a page describing a development workflow for Serve applications.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

architkulkarni · 2022-08-10T15:45:06Z

@shrekris-anyscale @edoakes @sihanwang41 @simon-mo This is still rough, but is ready for an initial content review.

I've left some questions and comments as "TODOs" that I need to resolve, but if you know the answer off the top of your head feel free to comment it. The code sample doesn't work at the moment, I think I'm doing something wrong with the ClassNode.

As a more general point, I don't know whether it's better to use serve.run(..bind(bind()) or to use dag.execute here. The two methods (dynamic dispatch with passing deployments into each other, vs. InputNode() call graph) don't seem compatible, in the sense that you wouldn't start with one of them for dev and switch to the other one for prod (you could, but you'd be rewriting a lot of code.)

doc/source/serve/dev-workflow.md

edoakes · 2022-08-10T18:01:04Z

doc/source/serve/dev-workflow.md

+
+Here we used the `execute` method to directly send a Python object to the deployment, rather than an HTTP request.
+
+% TODO: We didn't use serve.run() like we said at the begining; should we?


let's just use serve.run here instead of the dag execute thing for consistency. we can leave dag execute to the deployment graph docs

actually, maybe we should just not use deployment graph at all here and make it a simpler two-deployment app?

actually, maybe we should just not use deployment graph at all here and make it a simpler two-deployment app?

I think this is the key question, or at least the main thing I'm confused about. Are we saying that an expected part of the dev to prod transition is rewriting the code to go from serve.run to using a deployment graph?

How do you actually make a two-deployment pipeline app without using a deployment graph? You'd need to pass in the second deployment in a constructor, right? But to then later convert it to a graph, you'd need to rewrite all the deployment code to remove the deployments from the constructors and use InputNode, which seems like a lot of work.

Another perspective, will using serve.run in dev slow down the velocity or execute will bring other benefits ? If not, I don't think we need to provide the execute information in the doc.

edoakes · 2022-08-10T18:01:32Z

doc/source/serve/dev-workflow.md

+You can use the `serve run` CLI command to run and test your application locally using HTTP to send requests (similar to how you might use `uvicorn run` if you're familiar with Uvicorn):
+
+```bash
+serve run local_dev:HelloDeployment


would be nice to show the relevant info that gets printed to the console when you run this.

let's also mention that logs will be streamed. maybe make the deployment print something and show the output when you query it?

sihanwang41 · 2022-08-10T19:25:06Z

doc/source/serve/dev-workflow.md

+
+Let's see a simple example.
+
+```python


move the code to the doc_code dir?

sihanwang41 · 2022-08-10T19:35:28Z

doc/source/serve/dev-workflow.md

+
+Here we used the `execute` method to directly send a Python object to the deployment, rather than an HTTP request.
+
+% TODO: We didn't use serve.run() like we said at the begining; should we?


Another perspective, will using serve.run in dev slow down the velocity or execute will bring other benefits ? If not, I don't think we need to provide the execute information in the doc.

doc/source/serve/dev-workflow.md

Co-authored-by: Edward Oakes <[email protected]>

doc/source/serve/dev-workflow.md

shrekris-anyscale

Nice work so far! I've added some comments. I think it would be helpful to include info in each section about why or why not a user might want to use that development style. If it's a short reason, we could include that info directly in the header, so users don't have to read info that isn't relevant to them.

doc/source/serve/dev-workflow.md

Co-authored-by: shrekris-anyscale <[email protected]>

edoakes · 2022-08-11T02:02:39Z

@architkulkarni you have DCO and lint failures

…i/ray into serve-dev-workflow

…e-dev-workflow

doc/source/serve/dev-workflow.md

stephanie-wang · 2022-08-11T19:56:46Z

doc/source/serve/dev-workflow.md

+
+When making the transition from your local machine to a remote cluster, you'll need to make sure your cluster has a similar environment to your local machine--files, environment variables, and Python packages, for example. During development, you can use {ref}`Runtime Environments<runtime-environments>` to manage this in a flexible way.
+
+See [Ray Client](ray-client-under-construction) for more information on the Ray address specified here by the `--address` parameter.


It would be good to just explain how to build the address inline instead of linking to another guide. Also, is there a way to run this without Ray Client?

Under the hood, --address is passed to ray.init() on the laptop. If you pass in a ray:https:// address, it will use Ray Client. You could also specify a local Ray address, but for simplicity I'll leave that out of the guide.

doc/source/serve/dev-workflow.md

stephanie-wang · 2022-08-11T20:00:35Z

doc/source/serve/dev-workflow.md

+Let's see a simple example.
+
+```bash
+serve run --address=ray:https://<cluster-ip-address>:10001 --runtime-env-json='{"env_vars": {"MY_ENV_VAR": "my-value"}, "working_dir": "./project/src", "pip": ["requests", "chess"]}' local_dev:HelloDeployment


Can we split this example into multiple steps to make it more digestible? Concrete suggestion:

First explain how to run Serve on a remote cluster, no runtime env involved.

Then show the common case that's currently mentioned at the end: "A common pattern is to use the root directory of your project as the working_dir of your runtime environment when testing on a remote cluster."

End with the advanced case where a full runtime env json / yaml is used.

For step 1, I don't think there's a dev workflow that doesn't use runtime_env, because we need runtime_env to get the local code onto the cluster.

Why don't I try this ordering to make it more digestible:

Just use --working-dir

Use a full --runtime-env-json

Mention runtime_env YAML

Ah I see. Yes, that flow sounds good, thanks!

I gave this a shot, let me know how it looks

Co-authored-by: Stephanie Wang <[email protected]>

architkulkarni · 2022-08-11T20:49:26Z

@stephanie-wang Thanks for the review! Addressed/responded to the comments, could you please take another look?

…i/ray into serve-dev-workflow

stephanie-wang · 2022-08-11T21:12:41Z

doc/source/serve/dev-workflow.md

+When making the transition from your local machine to a remote cluster, you'll need to make sure your cluster has a similar environment to your local machine--files, environment variables, and Python packages, for example. During development, you can use {ref}`Runtime Environments<runtime-environments>` to manage these dependencies.
+
+Let's see a simple example. Run the following command on your local machine, with your remote cluster head node IP address substituted for `<head-node-ip-address>` in the command:
+
+```bash
+serve run --address=ray:https://<head-node-ip-address>:10001 --working_dir="./project/src" local_dev:HelloDeployment
+```
+
+This will connect to the remote cluster via Ray Client, upload the `working_dir` directory, and run your serve application. Here, the local directory specified by `working_dir` must contain `local_dev.py` so that it can be uploaded to the cluster and imported by Ray Serve.
+
+Once this is up and running, we can send requests to the application:
+
+```bash
+curl -X PUT http:https://<head-node-ip-address>:8000/?name=Ray
+# Hello, Ray! Hello, Ray!
+```
+
+If your runtime environment contains fields other than `working_dir`, use the `--runtime-env-json` argument:


Suggested change

When making the transition from your local machine to a remote cluster, you'll need to make sure your cluster has a similar environment to your local machine--files, environment variables, and Python packages, for example. During development, you can use {ref}`Runtime Environments<runtime-environments>` to manage these dependencies.

Let's see a simple example. Run the following command on your local machine, with your remote cluster head node IP address substituted for `<head-node-ip-address>` in the command:

```bash

serve run --address=ray:https://<head-node-ip-address>:10001 --working_dir="./project/src" local_dev:HelloDeployment

```

This will connect to the remote cluster via Ray Client, upload the `working_dir` directory, and run your serve application. Here, the local directory specified by `working_dir` must contain `local_dev.py` so that it can be uploaded to the cluster and imported by Ray Serve.

Once this is up and running, we can send requests to the application:

```bash

curl -X PUT http:https://<head-node-ip-address>:8000/?name=Ray

# Hello, Ray! Hello, Ray!

```

If your runtime environment contains fields other than `working_dir`, use the `--runtime-env-json` argument:

When making the transition from your local machine to a remote cluster, you'll need to make sure your cluster has a similar environment to your local machine--files, environment variables, and Python packages, for example.

Let's see a simple example that just packages the code. Run the following command on your local machine, with your remote cluster head node IP address substituted for `<head-node-ip-address>` in the command:

```bash

serve run --address=ray:https://<head-node-ip-address>:10001 --working_dir="<directory containing local_dev.py>" local_dev:HelloDeployment

This will connect to the remote cluster via Ray Client, upload the directory specified by --working_dir, and run your Serve application in this directory. We can make local_dev.py importable by Ray Serve by making sure that it is contained in the directory passed to --working-dir.

Once this is up and running, we can send requests to the application:

curl -X PUT http:https://<head-node-ip-address>:8000/?name=Ray # Hello, Ray! Hello, Ray!

For more complex dependencies, including files outside the working directory, environment variables, and Python packages, you can use {ref}Runtime Environments<runtime-environments> to manage these dependencies. Here is an example using the --runtime-env-json argument:

Oh no, I think my suggestion got mis-formatted by GitHub 🗡️

But basically I was trying to:

move the mention of runtime envs closer to where it's actually used in a command (for a first-time user, I don't think it will be obvious that --working-dir is just syntactic sugar for a runtime env)

explain semantics of --working-dir

Makes sense!

stephanie-wang

Thanks for the changes, it looks great! Left one more suggestion.

…e-dev-workflow

Adds a page describing a development workflow for Serve applications. Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: shrekris-anyscale <[email protected]> Co-authored-by: Stephanie Wang <[email protected]>

Adds a page describing a development workflow for Serve applications. Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: shrekris-anyscale <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Signed-off-by: Stefan van der Kleij <[email protected]>

Adds a page describing a development workflow for Serve applications. Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: shrekris-anyscale <[email protected]> Co-authored-by: Stephanie Wang <[email protected]>

Adds a page describing a development workflow for Serve applications. Co-authored-by: Edward Oakes <[email protected]> Co-authored-by: shrekris-anyscale <[email protected]> Co-authored-by: Stephanie Wang <[email protected]> Signed-off-by: Artur Niederfahrenhorst <[email protected]>

Add dev workflow page

9b95fdd

architkulkarni requested review from edoakes, simon-mo, jiaodong, shrekris-anyscale, sihanwang41 and a team as code owners August 10, 2022 15:39

architkulkarni assigned sihanwang41, edoakes, simon-mo and shrekris-anyscale Aug 10, 2022

edoakes reviewed Aug 10, 2022

View reviewed changes

edoakes added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 10, 2022

sihanwang41 reviewed Aug 10, 2022

View reviewed changes

edoakes reviewed Aug 10, 2022

View reviewed changes

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

Update doc/source/serve/dev-workflow.md

69a5dc6

Co-authored-by: Edward Oakes <[email protected]>

architkulkarni commented Aug 10, 2022

View reviewed changes

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

shrekris-anyscale reviewed Aug 10, 2022

View reviewed changes

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

doc/source/serve/dev-workflow.md Outdated Show resolved Hide resolved

architkulkarni and others added 3 commits August 10, 2022 15:45

Update doc/source/serve/dev-workflow.md

efd1b90

Co-authored-by: shrekris-anyscale <[email protected]>

Fix typo

d422a3c

Remove "project structure" and references to it

4308028

architkulkarni added 7 commits August 11, 2022 08:39

Merge branch 'serve-dev-workflow' of https://github.com/architkulkarn…

3db92af

…i/ray into serve-dev-workflow

Merge branch 'master' of https://github.com/ray-project/ray into serv…

a275707

…e-dev-workflow

Mention blocking and Ctrl-C

d5734f5

Add info about working dir contents

0f7366b

Add example of py module

c106321

Fix doc build, fix LINT

305011a

Don't say "doesn't test multinode"

d64b760

stephanie-wang requested changes Aug 11, 2022

View reviewed changes

stephanie-wang added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 11, 2022

stephanie-wang self-assigned this Aug 11, 2022

architkulkarni and others added 11 commits August 11, 2022 13:20

Update doc/source/serve/dev-workflow.md

0ccac78

Co-authored-by: Stephanie Wang <[email protected]>

Update doc/source/serve/dev-workflow.md

89d270c

Co-authored-by: Stephanie Wang <[email protected]>

Update doc/source/serve/dev-workflow.md

953df0c

Co-authored-by: Stephanie Wang <[email protected]>

Update doc/source/serve/dev-workflow.md

fd80994

Co-authored-by: Stephanie Wang <[email protected]>

Update doc/source/serve/dev-workflow.md

0d17209

Co-authored-by: Stephanie Wang <[email protected]>

Use bash comments to show output

680a9ae

Remove unused import

e632c01

Remove line about test code

8c18be4

Update doc/source/serve/dev-workflow.md

7c6d8e2

Co-authored-by: Stephanie Wang <[email protected]>

Reorder runtime env section

bf79994

Improve wording

f527dfc

architkulkarni removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 11, 2022

Merge branch 'serve-dev-workflow' of https://github.com/architkulkarn…

2ee0564

…i/ray into serve-dev-workflow

stephanie-wang reviewed Aug 11, 2022

View reviewed changes

stephanie-wang approved these changes Aug 11, 2022

View reviewed changes

stephanie-wang removed the copyediting-required label Aug 11, 2022

architkulkarni added 2 commits August 11, 2022 17:17

workign dir suggestion

2b29eeb

Merge branch 'master' of https://github.com/ray-project/ray into serv…

c7c80a6

…e-dev-workflow

edoakes merged commit 92e315f into ray-project:master Aug 12, 2022

edoakes mentioned this pull request Aug 12, 2022

[serve][docs] Development Workflow user guide #27446

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[serve][docs] Add dev workflow page #27746

[serve][docs] Add dev workflow page #27746

architkulkarni commented Aug 10, 2022

architkulkarni commented Aug 10, 2022

edoakes Aug 10, 2022

architkulkarni Aug 10, 2022

sihanwang41 Aug 10, 2022

edoakes Aug 10, 2022

sihanwang41 Aug 10, 2022

sihanwang41 Aug 10, 2022

shrekris-anyscale left a comment

edoakes commented Aug 11, 2022

stephanie-wang Aug 11, 2022

architkulkarni Aug 11, 2022

stephanie-wang Aug 11, 2022

architkulkarni Aug 11, 2022

stephanie-wang Aug 11, 2022

architkulkarni Aug 11, 2022

architkulkarni commented Aug 11, 2022

stephanie-wang Aug 11, 2022

stephanie-wang Aug 11, 2022

architkulkarni Aug 12, 2022

stephanie-wang left a comment


		Here we used the `execute` method to directly send a Python object to the deployment, rather than an HTTP request.

		% TODO: We didn't use serve.run() like we said at the begining; should we?


		When making the transition from your local machine to a remote cluster, you'll need to make sure your cluster has a similar environment to your local machine--files, environment variables, and Python packages, for example. During development, you can use {ref}`Runtime Environments<runtime-environments>` to manage this in a flexible way.

		See [Ray Client](ray-client-under-construction) for more information on the Ray address specified here by the `--address` parameter.

[serve][docs] Add dev workflow page #27746

[serve][docs] Add dev workflow page #27746

Conversation

architkulkarni commented Aug 10, 2022

Why are these changes needed?

Related issue number

Checks

architkulkarni commented Aug 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrekris-anyscale left a comment

Choose a reason for hiding this comment

edoakes commented Aug 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

architkulkarni commented Aug 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanie-wang left a comment

Choose a reason for hiding this comment