Failure with setup-ssh on Amazon Linux 2023 runners

[huydhn]

Earlier today, we rolled out Amazon Linux 2023 runners and this starts to cause setup-ssh to fail across the board. The change has been reverted https://github.com/pytorch-labs/pytorch-gha-infra/pull/417, and Amazon Linux 2 runners are being rolled out again. So, plz retry your failed CI jobs.

Root cause

The action https://github.com/pytorch/test-infra/blob/main/setup-ssh/src/main.ts#L89-L92 gets the current public IP of the runner by calling getEC2metadata, and if that fails, we call getIPs to api64.ipify.org to get the same information. On amzn2023, IMDSv2 is the only option to query ec2 medata https://aws.amazon.com/blogs/security/get-the-full-benefits-of-imdsv2-and-disable-imdsv1-across-your-aws-infrastructure/, so getEC2metadata always fails and the fallback to api64.ipify.org changes from rarely used to always being used, which causes rate limit. Testing on canary didn’t reveal anything because it didn’t have enough traffic to api64.ipify.org to be throttle

[huydhn]

Another anecdote on Amazon Linux 2023 upgrade, I suspect that some distributed NCCL tests are failing there, i.e. https://github.com/pytorch/pytorch/actions/runs/9600703357/job/26478744763#step:20:3099. A quick look at the code https://github.com/pytorch/pytorch/blob/main/test/distributed/test_c10d_nccl.py#L4023-L4028 seems to say that a timed out process now returns None instead of SIGABRT (6).

cc @jeanschmidt in case you see the failure on canary.