[Cluster launcher] botocore is a dependency that isn't automatically installed by Ray

[architkulkarni]

What happened + What you expected to happen

Installing the Ray nightly wheel should include all dependencies. However this isn't the case

pip install "ray[default]@https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp39-cp39-macosx_11_0_arm64.whl"

ray up

This leads to a modulenotfounderror for botocore. The workaround is to pip install boto3 but ideally it should be automatically installed. (A separate issue -- once this is fixed, we should also print a friendly error message if the user forgot to install ray[default] and instead just installed ray.)

Versions / Dependencies

Mac M1 python 3.9

Reproduction script

above

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[architkulkarni]

@hongchaodeng could you please post the exact traceback here for the botocore not found error?

[hongchaodeng]

Here's the traceback:

ray up python/ray/autoscaler/aws/example-minimal.yaml
hello
Cluster: aws-example-minimal

2023-09-28 10:16:30,801	INFO util.py:375 -- setting max workers for head node type to 0
2023-09-28 10:16:30,801	INFO util.py:379 -- setting max workers for ray.worker.default to 2
Traceback (most recent call last):
  File "/Users/hongchao.deng/miniforge3/envs/ray/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/scripts/scripts.py", line 2499, in main
    return cli()
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper
    return f(*args, **kwargs)
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/scripts/scripts.py", line 1299, in up
    create_or_update_cluster(
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/autoscaler/_private/commands.py", line 311, in create_or_update_cluster
    config = _bootstrap_config(config, no_config_cache=no_config_cache)
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/autoscaler/_private/commands.py", line 383, in _bootstrap_config
    provider_cls = importer(config["provider"])
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/autoscaler/_private/providers.py", line 29, in _import_aws
    from ray.autoscaler._private.aws.node_provider import AWSNodeProvider
  File "/Users/hongchao.deng/miniforge3/envs/ray/lib/python3.9/site-packages/ray/autoscaler/_private/aws/node_provider.py", line 9, in <module>
    import botocore
ModuleNotFoundError: No module named 'botocore'

[BwL1289]

@architkulkarni @hongchaodeng not sure this is the place to ask.

We already have boto3 and botocore installed and don't want them to be installed again while launching the cluster. Is there a way to skip installation on cluster launch?

Also, somewhat related, we get a .bashrc no file or directory on launch. Not sure what the cluster is attempting to do with it, but do you know of a way to rectify this as well?

[BwL1289]

Nevermind, I figured it out both of the above. My apologies.