Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Dashboard] Disable tune module if pandas version is incorrect #14381

Merged

Conversation

fyrestone
Copy link
Contributor

Why are these changes needed?

  • Problem
2021-02-26 11:36:35,653 WARNING reporter_agent.py:31 -- Install gpustat with 'pip install gpustat' to enable GPU monitoring.
2021-02-26 11:36:35,831 ERROR dashboard.py:236 -- The dashboard on node dooku.inc.alipay.net failed with the following error:
Traceback (most recent call last):
  File "/home/admin/ray/python/ray/new_dashboard/dashboard.py", line 222, in <module>
    loop.run_until_complete(dashboard.run())
  File "/home/admin/.pyenv/versions/3.6.5/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/home/admin/ray/python/ray/new_dashboard/dashboard.py", line 118, in run
    await self.dashboard_head.run()
  File "/home/admin/ray/python/ray/new_dashboard/head.py", line 190, in run
    modules = self._load_modules()
  File "/home/admin/ray/python/ray/new_dashboard/head.py", line 128, in _load_modules
    dashboard_utils.DashboardHeadModule)
  File "/home/admin/ray/python/ray/new_dashboard/utils.py", line 208, in get_all_modules
    importlib.import_module(name)
  File "/home/admin/.pyenv/versions/3.6.5/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/admin/ray/python/ray/new_dashboard/modules/tune/tune_head.py", line 13, in <module>
    from ray.tune import Analysis
  File "/home/admin/ray/python/ray/tune/__init__.py", line 2, in <module>
    from ray.tune.tune import run_experiments, run
  File "/home/admin/ray/python/ray/tune/tune.py", line 12, in <module>
    from ray.tune.analysis import ExperimentAnalysis
  File "/home/admin/ray/python/ray/tune/analysis/__init__.py", line 1, in <module>
    from ray.tune.analysis.experiment_analysis import ExperimentAnalysis, Analysis
  File "/home/admin/ray/python/ray/tune/analysis/experiment_analysis.py", line 21, in <module>
    from ray.tune.trial import Trial
  File "/home/admin/ray/python/ray/tune/trial.py", line 27, in <module>
    from ray.tune.utils.trainable import TrainableUtil
  File "/home/admin/ray/python/ray/tune/utils/trainable.py", line 7, in <module>
    import pandas as pd
  File "/home/admin/.pyenv/versions/3.6.5/envs/ray/lib/python3.6/site-packages/pandas/__init__.py", line 55, in <module>
    from pandas.core.api import (
  File "/home/admin/.pyenv/versions/3.6.5/envs/ray/lib/python3.6/site-packages/pandas/core/api.py", line 29, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/home/admin/.pyenv/versions/3.6.5/envs/ray/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/home/admin/.pyenv/versions/3.6.5/envs/ray/lib/python3.6/site-packages/pandas/core/groupby/generic.py", line 56, in <module>
    import pandas.core.algorithms as algorithms
AttributeError: module 'pandas' has no attribute 'core'

Reproduce steps:

  1. Python installed an incompatible version of pandas
  2. pip install ray (ray does not require pandas)
  3. ray.init() failed with above error

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@fyrestone fyrestone added the dashboard Issues specific to the Ray Dashboard label Feb 26, 2021
@kfstorm
Copy link
Member

kfstorm commented Feb 26, 2021

Shouldn't we skip loadding dashboard/modules/tune/ when ray[tune] is not installed?

@fyrestone
Copy link
Contributor Author

Shouldn't we skip loadding dashboard/modules/tune/ when ray[tune] is not installed?

Then the API GET /tune/availability will be unavailable. We can add an API to the dashboard to query the status of all modules in the future.

@kfstorm
Copy link
Member

kfstorm commented Feb 26, 2021

I see that we've already added try-except for ImportError.

I don't think this is a correct fix. The error @WangTaoTheTonic saw is environment-specific, not a Ray bug. What if @WangTaoTheTonic saw another exception tomorrow?

@fyrestone
Copy link
Contributor Author

I see that we've already added try-except for ImportError.

I don't think this is a correct fix. The error @WangTaoTheTonic saw is environment-specific, not a Ray bug. What if @WangTaoTheTonic saw another exception tomorrow?

The exception AttributeError: module 'pandas' has no attribute 'core' may be caused by missing some libraries in the system because I can't reproduce it. Another fix is to except Exception instead of (ImportError, AttributeError).

@WangTaoTheTonic
Copy link
Contributor

The core issue is that wrong pandas(no matter caused by version or environment) impacted tests running which unrelated with tune/pandas.
I think it's ok for us to see this error when we run test module who needs pandas, but not for those who doesn't.

@fyrestone fyrestone added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Mar 1, 2021
@edoakes edoakes merged commit 3616424 into ray-project:master Mar 9, 2021
@kfstorm kfstorm deleted the disable_tune_module_if_import_failed branch March 9, 2021 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dashboard Issues specific to the Ray Dashboard tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants