[Core] runtime_env remote URI authentication issue

[Martin4R]

What happened + What you expected to happen

I tried to use remote URLs in working_dir and py_module params of runtime-environments to download zip files from a private GitLab package registry. I did this with the fruit-example of Ray Serve with KubeRay.

The way mentioned in the documentation is to use HTTP-Basic authentication directly in the URL (see https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#option-2-manually-create-url-slower-to-implement-but-recommended-for-production-environments).

The described approach from the docs does not work for me and also has some side effects:

• credentials are then visible in the K8s config, which one can read with read-access (also in kubectl describe RayService)
• credentials get printed in Kuberay K8s-operator log file
• username:password gets included in the folder-name in the file-system: /tmp/ray/session_latest/runtime-resources/working_dir_files/https_ray-cluster:realpwdwouldbehere@gitlab_com_api_v4_projects_42_packages_generic_fruit_example_b5b4a19a_fruit_example
• the download of the zip file works, but then python complains that module ‘fruit’ cannot be found. This is because the folder-name of before mentioned folder gets included in the environment variable PYTHONPATH, which uses “:” as separator (and username:password also contains a colon as separator). Therefore the folder-name is split in 2 parts and Python searches in the wrong place for the module.

To work around the issues, I used a .netrc file instead for authentication.

Versions / Dependencies

• Ray 2.0.0
• Kuberay 0.3.0
• Python 3.7

Reproduction script

I used the example of a Kuberay RayService from here: https://docs.ray.io/en/latest/serve/production-guide/kubernetes.html
I used this example RayService CR with an HTTP-Basic protected zip-file for the working_dir param.

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[shrekris-anyscale]

The importing issue should be resolved by #28250. The netrc workaround is likely best for production use-cases, so the username and password aren't exposed. We should follow up with documentation explaining how to use netrc with runtime_envs.

[Xalag]

One option for Kuberay:
Create a secret which has ".netrc" as a key and the contents of the .netrc-file as the value.
Then use it as a mounted volume and set the environment variable NETRC to the destination:

containers:
            - name: ray-head
              image: rayproject/ray:2.0.0-py39
              volumeMounts:
                - mountPath: "/home/ray/netrcvolume/"
                  name: netrc-kuberay
                  readOnly: true
              env:
                - name: NETRC
                  value: "/home/ray/netrcvolume/.netrc"
volumes:
            - name: netrc-kuberay
              secret:
                secretName: secret-netrc-kuberay

[simon-mo]

Closing as this has been answered by @shrekris-anyscale and thanks @Xalag for providing the code same in KubeRay