update grpc to 1.43

[acxz]

add patch for newer setuptools, can be removed once grpc 1.44 is release

Why are these changes needed?

With grpc updated to 1.43, one of the patches is not needed.

Patch needed when building locally for newer setuptools version. See grpc/grpc#28392 for more details.
Also needed as a prereq to #21221

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

Before we merge, can you share the update from 1.42 -> 1.43?

[acxz]

Before we merge, can you share the update from 1.42 -> 1.43?

I'm a bit confused by that. As in link to the changelog of grpc? Here it is: https://github.com/grpc/grpc/releases/tag/v1.43.2

If you are asking if any changes need to be made in how we use grpc, to be honest I did not perform a thorough check in the behavior before and after the update.

As for why I updated to 1.43.0 is because we can remove a patch we have used and the patch to add is built on top of the later code of grpc.

[mwtian]

Thanks for updating grpc!

[mwtian]

LGTM if tests pass.

[rkooo567]

I was asking the release notes! Thanks for the responde :)!

[bveeramani]

‼️ ACTION REQUIRED ‼️

We've switched our code formatter from YAPF to Black (see #21311).

To prevent issues with merging your code, here's what you'll need to do:

1. Install Black

pip install -I black==21.12b0

2. Format changed files with Black

curl -o format-changed.sh https://gist.githubusercontent.com/bveeramani/42ef0e9e387b755a8a735b084af976f2/raw/7631276790765d555c423b8db2b679fd957b984a/format-changed.sh
chmod +x ./format-changed.sh
./format-changed.sh
rm format-changed.sh

3. Commit your changes.

git add --all
git commit -m "Format Python code with Black"

4. Merge master into your branch.

git pull upstream master

5. Resolve merge conflicts (if necessary).

After running these steps, you'll have the updated format.sh.

[acxz]

@bveeramani n/a since this code has not python changes

[acxz]

@rkooo567 any updates on this? I believe the failing unit tests are not due to the fix. But if you can take a look and confirm that would be great! This is currently the bottleneck behind shipping py310 wheels.

[rkooo567]

Can you try merging the latest master? I've never seen failures from mac-apple-worker

[acxz]

@rkooo567 I have rebased master.

It looks like mac-apple-worker is failing because of improper fetching of certain codebases. Some URL's not found are seen in the error log. https://buildkite.com/ray-project/ray-builders-pr/builds/23464#f639a640-4a15-4866-9bb1-51261925ef91/41-1451

[rkooo567]

Sorry for the delay, but can you merge the master one more time to retrigger build?

The mac tests failed due to infra errors yesterday, but I think for this pr, I’d like to make sure all mac tests pass before merging it

[acxz]

Looks like there are test issues related to grpc concerning mac-apple-worker: https://buildkite.com/ray-project/ray-builders-pr/builds/23580#cfe4b8ea-dc82-46e6-8236-79369e313a8e/42-1681

I'll try to look into this.

[acxz]

I spent some time, but couldn't really decrypt the error into where the relevant source code could be. It would great if someone with some Mac expertise can take a look at this. Trying to search on the grpc repo about the no symbols warning doesn't give much results either.

[rkooo567]

cc @mwtian do you think you have some bandwidth to take a look? Otherwise, I can try next week

[mwtian]

The no symbols warnings should be benign here. //cpp:api_test passed locally with this PR on my macbook. I will rerun the tests a few times first.

[mwtian]

The actual error is dyld: malformed mach-o: load commands size (34504) > 32768, which I can reproduce in macos 10.15 on ec2. I'm not sure which shared library is problematic and how to fix it though. There are talks about using a shorter path when building the shared library, or using static linking. I will take another look later.

[rkooo567]

Also, @SongGuyang do you have any idea? The only failing test is from cpp APIs

[SongGuyang]

I will try to reproduce in my mac.

[mwtian]

I will try to reproduce in my mac.

Btw I'm unable to reproduce this on Monterey (12.1), but this is reproducible on 10.15 in the CI environment.

[SongGuyang]

Btw I'm unable to reproduce this on Monterey (12.1), but this is reproducible on 10.15 in the CI environment.

Thank you for this notice! I just updated my mac to 12.1 yesterday😭. @qicosmos can you help?

[SongGuyang]

The test also passed in my mac(12.1).

[qicosmos]

I hava tested on Mac10.15.5, will be an error: dyld: malformed mach-o: load commands size (34504) > 32768

I noticed the api_test linkstatic is False, others are True, i changed it to True, it works well, i will create a PR to solve the issue.

[rkooo567]

Thanks for taking a look at this guys!

[mwtian]

Can you merge master again? The test should have been fixed now (b8fbec1).

[acxz]

Looks like the post-wheels test are failing. It is interesting to note that they didn't fail before. Are those tests flaky or should we dig deeper into the issue?

[acxz]

Yay! All the tests pass!!! @rkooo567 I believe this can now finally be merged! haha

[mwtian]

Btw the post wheels test failure was a known flaky test which passed on rerun.

[rkooo567]

Thanks a lot! Btw, 1.44 was released today lol

[acxz]

Looool, I'll make a PR on this later today then just to get rid of the patch we have locally. But at least with this merged py310 unit tests can continue.