Including psutil & setproctitle #7031

ijrsvt · 2020-02-03T21:45:56Z

Why are these changes needed?

Including psutil & setproctitle into the distribution to allow for dashboard to be built automatically.

Related issue number

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.

AmplabJenkins · 2020-02-03T21:52:18Z

Can one of the admins verify this patch?

build.sh

simon-mo · 2020-02-03T22:20:34Z

python/ray/__init__.py

+ setproc_path = os.path.join(os.path.abspath(os.path.dirname(__file__)),
+ "setproctitle_files")
+ sys.path.insert(0, setproc_path)
+


We don't need mess with the import path if we are directly import from ray.thirdparty_files.psutil

simon-mo · 2020-02-03T22:21:14Z

python/ray/reporter.py

+ psutil_path = os.path.join(
+ os.path.abspath(os.path.dirname(__file__)), "psutil_files")
+ sys.path.insert(0, psutil_path)
+
 try:
 import psutil


Let's make sure to change all occurrence of import psutil and import setproctitle to import ray.thirdparty.psutil import psutil

python/setup.py

simon-mo · 2020-02-03T23:26:02Z

Tested locally, it worked!

AmplabJenkins · 2020-02-03T23:58:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21310/
Test FAILed.

AmplabJenkins · 2020-02-04T00:38:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21313/
Test FAILed.

AmplabJenkins · 2020-02-04T00:43:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21314/
Test FAILed.

AmplabJenkins · 2020-02-04T06:48:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21336/
Test FAILed.

AmplabJenkins · 2020-02-04T07:26:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21334/
Test FAILed.

AmplabJenkins · 2020-02-04T10:00:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21338/
Test PASSed.

AmplabJenkins · 2020-02-04T19:33:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21391/
Test FAILed.

AmplabJenkins · 2020-02-04T19:33:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21390/
Test FAILed.

.travis.yml

Co-Authored-By: Simon Mo <[email protected]>

AmplabJenkins · 2020-02-05T04:23:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21400/
Test PASSed.

python/ray/memory_monitor.py

python/ray/reporter.py

python/ray/tests/test_advanced_3.py

ci/travis/format.sh

simon-mo · 2020-02-05T18:43:13Z

ci/travis/install-dependencies.sh

@@ -61,3 +61,6 @@ if [[ "$PYTHON" == "3.5" ]] || [[ "$MAC_WHEELS" == "1" ]]; then
 source $HOME/.nvm/nvm.sh
 nvm install node
 fi
+
+pip install -q psutil setproctitle \
+ --target="$ROOT_DIR/../../python/ray/thirdparty_files"


simon-mo · 2020-02-05T18:44:08Z

python/ray/__init__.py

@@ -51,6 +51,11 @@
 os.path.abspath(os.path.dirname(__file__)), "pyarrow_files")
 sys.path.insert(0, pyarrow_path)

+psutil_path = os.path.join(


psutil_path -> thirdparty_path

Add a comment saying "This is done to make sure we use the bundled version of psutil and setproctitle"

simon-mo · 2020-02-05T18:44:20Z

python/ray/exceptions.py

- import setproctitle
-except ImportError:
- setproctitle = None
+import ray.thirdparty_files.setproctitle as setproctitle


import ray
import setproctitle

python/ray/worker.py

AmplabJenkins · 2020-02-05T19:57:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21444/
Test FAILed.

…uiring import ray

AmplabJenkins · 2020-02-05T20:44:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21445/
Test FAILed.

AmplabJenkins · 2020-02-06T01:18:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21450/
Test PASSed.

AmplabJenkins · 2020-02-06T01:53:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21454/
Test PASSed.

AmplabJenkins · 2020-02-06T02:52:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21452/
Test FAILed.

Ian Rodney added 12 commits January 29, 2020 13:38

Kill Actor UI addition

410f7d2

Formatted api.ts

133177e

Only killing if actor is alive

9953f36

Displaying only if actor is alive

aa96374

Initial Pieces

d46fbcf

Removing ray.psutil (buggy)

1d5ff04

Modifed build.sh

c373456

Cleaning up todos

a407d88

Formatting

c9bc3dd

Merge branch 'master' into BundleDeps

8d29970

Removing extra import

b55fab0

cleaning up reporter.py

4086288

Adding consistency

523d50e

ijrsvt requested a review from simon-mo February 3, 2020 22:00

Ian Rodney added 2 commits February 3, 2020 14:01

Adding consistency

83139f7

Merge branch 'BundleDeps' of github.com:ijrsvt/ray into BundleDeps

b73c9f7

simon-mo reviewed Feb 3, 2020

View reviewed changes

Ian Rodney added 2 commits February 3, 2020 14:35

making discussed changes

c35f0de

doing it the new way

9a837d1

simon-mo marked this pull request as ready for review February 3, 2020 23:25

Ian Rodney added 5 commits February 3, 2020 16:47

Fixing up some unresolved problems and minor formatting errors

ba9baba

Adding psutil to modules

d8763cd

reverting to old pattern of imports

b8c8a32

Formatting problems are resolved

d369d41

Formatting problems are resolved

e2a1897

ijrsvt added 2 commits February 4, 2020 10:28

Adding import ray to allow for docs to build

c8e7f96

Changing flake8 tests to be less picky

369280f

simon-mo reviewed Feb 4, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

Update .travis.yml

c7318cb

Co-Authored-By: Simon Mo <[email protected]>

robertnishihara mentioned this pull request Feb 5, 2020

Ray nodes should respect docker limits #7039

Merged

3 tasks

ijrsvt added 2 commits February 5, 2020 10:20

Adding psutil path to __init__ to help doc building

3197d03

Merge branch 'BundleDeps' of github.com:ijrsvt/ray into BundleDeps

c7bbb94

simon-mo reviewed Feb 5, 2020

View reviewed changes

Add import ordering checker

3f41b33

Changing to use single sys path modification in __init__ and then req…

e36885b

…uiring import ray

ijrsvt added 2 commits February 5, 2020 12:48

Adding YAPF excludes

9dda64a

Fixing linter errors and formatting changes

a904ddc

simon-mo approved these changes Feb 5, 2020

View reviewed changes

simon-mo merged commit 0826f95 into ray-project:master Feb 5, 2020

ijrsvt deleted the BundleDeps branch August 26, 2020 00:24

esquires mentioned this pull request Dec 10, 2020

remove debug extras from setup.py #12751

Merged

2 tasks

acxz mentioned this pull request Dec 10, 2020

Add support for Python 3.9 #12613

Merged

6 tasks

simon-mo mentioned this pull request Jul 16, 2021

vendor colorama #17058

Closed

6 tasks

adamjstewart mentioned this pull request Nov 5, 2021

Update py-ytopt package and add necessary dependencies spack/spack#26879

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Including psutil & setproctitle #7031

Including psutil & setproctitle #7031

ijrsvt commented Feb 3, 2020

AmplabJenkins commented Feb 3, 2020

simon-mo Feb 3, 2020

simon-mo Feb 3, 2020

simon-mo commented Feb 3, 2020

AmplabJenkins commented Feb 3, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 5, 2020

simon-mo Feb 5, 2020

simon-mo Feb 5, 2020

simon-mo Feb 5, 2020

AmplabJenkins commented Feb 5, 2020

AmplabJenkins commented Feb 5, 2020

AmplabJenkins commented Feb 6, 2020

AmplabJenkins commented Feb 6, 2020

AmplabJenkins commented Feb 6, 2020

Including psutil & setproctitle #7031

Including psutil & setproctitle #7031

Conversation

ijrsvt commented Feb 3, 2020

Why are these changes needed?

Related issue number

Checks

AmplabJenkins commented Feb 3, 2020

simon-mo Feb 3, 2020

Choose a reason for hiding this comment

simon-mo Feb 3, 2020

Choose a reason for hiding this comment

simon-mo commented Feb 3, 2020

AmplabJenkins commented Feb 3, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 4, 2020

AmplabJenkins commented Feb 5, 2020

simon-mo Feb 5, 2020

Choose a reason for hiding this comment

simon-mo Feb 5, 2020

Choose a reason for hiding this comment

simon-mo Feb 5, 2020

Choose a reason for hiding this comment

AmplabJenkins commented Feb 5, 2020

AmplabJenkins commented Feb 5, 2020

AmplabJenkins commented Feb 6, 2020

AmplabJenkins commented Feb 6, 2020

AmplabJenkins commented Feb 6, 2020