[dashboard] Fixes dashboard issues when environments have set http_proxy

[ConeyLiu]

Why are these changes needed?

Fixes dashboard issues when environments have set http_proxy. Without this patch:

In [1]: import ray

In [2]: address_info = ray.init(include_dashboard=True)
2020-12-03 15:45:03,102 INFO services.py:1171 -- View the Ray dashboard at http:https://127.0.0.1:8265

In [3]: 2020-12-03 15:45:05,846 WARNING worker.py:1011 -- The agent on node xyliu failed with the following error:
Traceback (most recent call last):
  File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 300, in <module>
    loop.run_until_complete(agent.run())
  File "/home/lxy/miniconda3/envs/ray/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 174, in run
    agent_ip_address=self.ip))
  File "/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/grpc/experimental/aio/_call.py", line 285, in __await__
    self._cython_call._status)
grpc.experimental.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "{"created":"@1606981505.836508871","description":"Error received from peer ipv4:10.239.4.100:913","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Stream removed","grpc_status":2}"
>

(pid=raylet) Traceback (most recent call last):
(pid=raylet)   File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 310, in <module>
(pid=raylet)     raise e
(pid=raylet)   File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 300, in <module>
(pid=raylet)     loop.run_until_complete(agent.run())
(pid=raylet)   File "/home/lxy/miniconda3/envs/ray/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
(pid=raylet)     return future.result()
(pid=raylet)   File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 174, in run
(pid=raylet)     agent_ip_address=self.ip))
(pid=raylet)   File "/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/grpc/experimental/aio/_call.py", line 285, in __await__
(pid=raylet)     self._cython_call._status)
(pid=raylet) grpc.experimental.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
(pid=raylet)    status = StatusCode.UNKNOWN
(pid=raylet)    details = "Stream removed"
(pid=raylet)    debug_error_string = "{"created":"@1606981505.836508871","description":"Error received from peer ipv4:10.239.4.100:913","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Stream removed","grpc_status":2}"
(pid=raylet) >
In [3]: 2020-12-03 15:45:07,709 WARNING worker.py:1011 -- The agent on node xyliu failed with the following error:
Traceback (most recent call last):
  File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 300, in <module>
    loop.run_until_complete(agent.run())
  File "/home/lxy/miniconda3/envs/ray/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/lxy/git_repository/ray/python/ray/new_dashboard/agent.py", line 174, in run
    agent_ip_address=self.ip))
  File "/home/lxy/miniconda3/envs/ray/lib/python3.7/site-packages/grpc/experimental/aio/_call.py", line 285, in __await__
    self._cython_call._status)
grpc.experimental.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "{"created":"@1606981507.704827237","description":"Error received from peer ipv4:10.239.4.100:913","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Stream removed","grpc_status":2}"
>

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ConeyLiu]

Hi @mfitton, is there any way to trigger the dashboard UT tests locally?

[mfitton]

Hey @ConeyLiu, from your ray directory you can run pytest dashboard/tests to execute the top-level unit-tests and also pytest dashboard/modules/MODULE_NAME/tests for tests in individual modules.

[ConeyLiu]

Hi @mfitton, I got the following errors:

ImportError while loading conftest '/Users/xianyang/git/ray/dashboard/tests/conftest.py'.
dashboard/tests/conftest.py:3: in <module>
    from ray.tests.conftest import *  # noqa
E   ModuleNotFoundError: No module named 'ray.tests'

With bazel test python/ray/new_dashboard/..., I got similar errors.

[mfitton]

@ConeyLiu hm, strange. Okay, well the dashboard code is symlinked to ray/python/ray/new_dashboard, so if you use that as the base rather than ray/dashboard as I had mentioned above, it should definitely be able to find the conftest file, as it'll be in the same hierarchy.

Not sure why running them at the other level doesn't work in your install, sorry for the inconvenience.

[ConeyLiu]

Hi @mfitton, thanks for the help.

[fyrestone]

It's cleaner to set an option here instead of providing a special channel. If the existing API cannot meet our requirements, then we create a new API.

options = (("grpc.enable_http_proxy", 0), )
channel = aiogrpc.insecure_channel(address, options=options)

@rkooo567 May be these two lines are enough?

[ConeyLiu]

I am OK to change it back. And create_insecure_channel is just a wrapper function.

[rkooo567]

cc @fyrestone Please ping me after you approve the PR!

[fyrestone]

LGTM