update grpc to 1.44.0 #22384

acxz · 2022-02-15T13:43:10Z

Why are these changes needed?

Updates grpc to 1.44.0 to remove local patch needed for grpc to build.

EDIT: there have been changes to how python is found (mostly removing python2 support) and as such the local python-patch we have for grpc needs to be modified.

This time contributing it to upstream (grpc/grpc#28895) so that it'll get added in a newer version!

For anyone that comes across this:
Here is the error itself for why we need the grpc-python.patch file: https://buildkite.com/ray-project/ray-builders-pr/builds/24659#d293616f-225d-41f9-8de2-03780f12b13f/2386-2416

Related issue number

Similar in spirit to #21866

GRPC changelog: https://github.com/grpc/grpc/releases/tag/v1.44.0

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Relevant People:
@rkooo567

rkooo567 · 2022-02-16T07:46:20Z

Can you merge the latest master?

acxz · 2022-02-17T19:07:21Z

@rkooo567 should I rebase from master again or are the tests we see known to be flaky?

mwtian · 2022-02-18T05:36:42Z

Test failures should be unrelated.

rkooo567 · 2022-02-18T12:42:38Z

@mwtian is the Mac cpp test unrelated? It seems a bit weird to me

mwtian · 2022-02-19T00:58:54Z

Yeah, I didn't see the Mac C++ tests earlier. Their failures seem problematic. I will be OOO next week. If we haven't fixed it then, I can take a look.

acxz · 2022-03-03T22:20:02Z

@mwtian sorry to ping, but if you can glance at the mac test or have anyone else that has a mac machine test it out that would be wonderful.

mwtian · 2022-03-03T22:24:57Z

@rkooo567 mentioned he will take a look, but I can give it a try tomorrow too.

rkooo567 · 2022-03-04T00:40:05Z

Yeah I had no time for this (maybe until the end of next week)

mwtian · 2022-03-04T01:41:12Z

This is the same issue as when we upgraded grpc last time. Now it is happening for all tests.

dyld: malformed mach-o: load commands size (32912) > 32768

I think we should upgrade the CI MacOS version higher then 10.15 Catalina.

mwtian · 2022-03-04T19:50:33Z

I created a workaround in #22829. @acxz feel free to try again after that is merged, or patch in the change and rerun the CI.

There are problems with running C++ tests in MacOS 10.15 Catalina, when upgrading to the newest grpc due to dynamic linking: #22384 (comment). The problem does not exist for Python tests in Catalina, or in C++ tests of other systems. Upgrading MacOS CI from Catalina is also blocked in the short term: ray-project/reef#24 (comment) So working around the issue by using static linking for C++ tests on Mac.

acxz · 2022-03-05T05:21:36Z

Thanks @mwtian I hadn't realized there was a blocker on updating the macos ci version or else I wouldn't have pushed so hard for this, especially considering this PR itself is not needed for getting py3.10 support (which is what I'm currently trying to resolve)

acxz · 2022-03-05T05:22:40Z

Oh just saw it got merged, let me rebase to restart the ci on this PR.

mwtian

All tests passed, after rerunning one Windows test failure.

rkooo567 · 2022-03-07T12:53:54Z

Thanks again for the patch!

mwtian · 2022-03-09T17:49:55Z

Seeing these errors for building Windows wheels on master, which somehow did not appear during CI for this PR: https://buildkite.com/ray-project/ray-builders-branch/builds/6402#ffee35fd-ab9b-486d-bd7f-91ae876cddbf/2600-2623

(16:46:20) INFO: Repository local_config_python instantiated at:
--
  | C:/install/ray/WORKSPACE:9:19: in <toplevel>
  | C:/install/ray/bazel/ray_deps_build_all.bzl:17:12: in ray_deps_build_all
  | C:/tmp/4lhdprva/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:459:21: in grpc_deps
  | C:/tmp/4lhdprva/external/com_github_grpc_grpc/bazel/grpc_python_deps.bzl:61:21: in grpc_python_deps
  | Repository rule python_configure defined at:
  | C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl:354:35: in <toplevel>
  | (16:46:20) ERROR: An error occurred during the fetch of repository 'local_config_python':
  | Traceback (most recent call last):
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 344, column 35, in _python_autoconf_impl
  | _create_single_version_package(
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 291, column 33, in _create_single_version_package
  | python_bin = _get_python_bin(repository_ctx, bin_path_key, default_bin_path, allow_absent)
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 165, column 14, in _get_python_bin
  | _fail("Cannot find python in PATH, please make sure " +
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 33, column 9, in _fail
  | fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
  | Error in fail: Python Configuration Error: Cannot find python in PATH, please make sure python is installed and add its directory in PATH, or --define PYTHON3_BIN_PATH='/something/else'.
  | PATH=C:\openjdk-16\bin;C:\Program Files\Eclipse Foundation\jdk-8.0.302.8-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\ProgramData\Chocolatey\bin;C:\Program Files\PowerShell\7;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;C:\Users\ContainerAdministrator\.dotnet\tools;C:\Miniconda3;C:\Miniconda3\Scripts;C:\Miniconda3\Library\bin;C:\Program Files\Git\cmd;C:\bazel;C:\msys64\mingw64\bin;C:\msys64\usr\bin

(16:46:20) ERROR: Error fetching repository: Traceback (most recent call last):
--
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 344, column 35, in _python_autoconf_impl
  | _create_single_version_package(
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 291, column 33, in _create_single_version_package
  | python_bin = _get_python_bin(repository_ctx, bin_path_key, default_bin_path, allow_absent)
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 165, column 14, in _get_python_bin
  | _fail("Cannot find python in PATH, please make sure " +
  | File "C:/tmp/4lhdprva/external/com_github_grpc_grpc/third_party/py/python_configure.bzl", line 33, column 9, in _fail
  | fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
  | Error in fail: Python Configuration Error: Cannot find python in PATH, please make sure python is installed and add its directory in PATH, or --define PYTHON3_BIN_PATH='/something/else'.
  | PATH=C:\openjdk-16\bin;C:\Program Files\Eclipse Foundation\jdk-8.0.302.8-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\ProgramData\Chocolatey\bin;C:\Program Files\PowerShell\7;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;C:\Users\ContainerAdministrator\.dotnet\tools;C:\Miniconda3;C:\Miniconda3\Scripts;C:\Miniconda3\Library\bin;C:\Program Files\Git\cmd;C:\bazel;C:\msys64\mingw64\bin;C:\msys64\usr\bin

(16:46:20) ERROR: BUILD.bazel:2354:12: //:python/ray/_raylet.so depends on @local_config_python//:python_headers in repository @local_config_python which failed to fetch. no such package '@local_config_python//': Python Configuration Error: Cannot find python in PATH, please make sure python is installed and add its directory in PATH, or --define PYTHON3_BIN_PATH='/something/else'.
--
  | PATH=C:\openjdk-16\bin;C:\Program Files\Eclipse Foundation\jdk-8.0.302.8-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\ProgramData\Chocolatey\bin;C:\Program Files\PowerShell\7;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;C:\Users\ContainerAdministrator\AppData\Local\Microsoft\WindowsApps;C:\Users\ContainerAdministrator\.dotnet\tools;C:\Miniconda3;C:\Miniconda3\Scripts;C:\Miniconda3\Library\bin;C:\Program Files\Git\cmd;C:\bazel;C:\msys64\mingw64\bin;C:\msys64\usr\bin

(16:46:20) WARNING: errors encountered while analyzing target '//:ray_pkg.out': it will not be built
......

This reverts commit 5ebc32d.

acxz · 2022-03-09T18:03:11Z

Riiip, let me open up a PR and test out a patch. The solution is probably just a simple change from python to python.exe. But I am a bit worried that CI passed.

mwtian · 2022-03-09T18:07:41Z

Great, thanks for looking into this and fixing the local path!

This reverts commit 5ebc32d.

rkooo567 approved these changes Feb 16, 2022

View reviewed changes

rkooo567 assigned rkooo567 and mwtian Feb 16, 2022

mwtian approved these changes Feb 16, 2022

View reviewed changes

mwtian mentioned this pull request Mar 4, 2022

[CI] run Mac C++ tests with static linking #22829

Merged

6 tasks

update grpc to 1.44.0

d750a3f

mwtian approved these changes Mar 7, 2022

View reviewed changes

rkooo567 merged commit 5ebc32d into ray-project:master Mar 7, 2022

acxz deleted the grpc-update branch March 7, 2022 15:07

mwtian added a commit that referenced this pull request Mar 9, 2022

Revert "[Core] Update grpc to 1.44.0 (#22384)"

8aaa445

This reverts commit 5ebc32d.

mwtian mentioned this pull request Mar 9, 2022

Revert "update grpc to 1.44.0" #22958

Merged

simon-mo pushed a commit that referenced this pull request Mar 9, 2022

Revert "[Core] Update grpc to 1.44.0 (#22384)" (#22958)

3ccc2aa

This reverts commit 5ebc32d.

acxz mentioned this pull request Mar 16, 2022

update grpc to 1.48.0 #23246

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update grpc to 1.44.0 #22384

update grpc to 1.44.0 #22384

acxz commented Feb 15, 2022 •

edited

Loading

rkooo567 commented Feb 16, 2022

acxz commented Feb 17, 2022

mwtian commented Feb 18, 2022

rkooo567 commented Feb 18, 2022

mwtian commented Feb 19, 2022

acxz commented Mar 3, 2022

mwtian commented Mar 3, 2022

rkooo567 commented Mar 4, 2022

mwtian commented Mar 4, 2022

mwtian commented Mar 4, 2022

acxz commented Mar 5, 2022 •

edited

Loading

acxz commented Mar 5, 2022

mwtian left a comment

rkooo567 commented Mar 7, 2022

mwtian commented Mar 9, 2022 •

edited

Loading

acxz commented Mar 9, 2022

mwtian commented Mar 9, 2022

update grpc to 1.44.0 #22384

update grpc to 1.44.0 #22384

Conversation

acxz commented Feb 15, 2022 • edited Loading

Why are these changes needed?

Related issue number

Checks

rkooo567 commented Feb 16, 2022

acxz commented Feb 17, 2022

mwtian commented Feb 18, 2022

rkooo567 commented Feb 18, 2022

mwtian commented Feb 19, 2022

acxz commented Mar 3, 2022

mwtian commented Mar 3, 2022

rkooo567 commented Mar 4, 2022

mwtian commented Mar 4, 2022

mwtian commented Mar 4, 2022

acxz commented Mar 5, 2022 • edited Loading

acxz commented Mar 5, 2022

mwtian left a comment

Choose a reason for hiding this comment

rkooo567 commented Mar 7, 2022

mwtian commented Mar 9, 2022 • edited Loading

acxz commented Mar 9, 2022

mwtian commented Mar 9, 2022

acxz commented Feb 15, 2022 •

edited

Loading

acxz commented Mar 5, 2022 •

edited

Loading

mwtian commented Mar 9, 2022 •

edited

Loading