Time check retried too fast

[scottyeager]

This loop seems to run without any wait before retry on this request:

zos/pkg/perf/healthcheck/ntp.go

Line 55 in 928723d

timeRes, err := http.Get("https://worldtimeapi.org/api/timezone/UTC")

Then sometimes this can start before the node's network is fully ready and cause about 3000 retries in two seconds. See for example this section of logs from node 6589 starting at 2024-07-18T08:19:50Z:

[+] noded: 2024-07-18T08:19:51Z error failed to run ntp check error="failed to get date: Get \"https://worldtimeapi.org/api/timezone/UTC\": dial tcp: lookup worldtimeapi.org on 192.168.6.1:53: dial udp 192.168.6.1:53: connect: network is unreachable"
... (3000 lines of mostly duplicate errors omitted)
[+] noded: 2024-07-18T08:19:50Z error failed to run ntp check error="failed to get date: Get \"https://worldtimeapi.org/api/timezone/UTC\": dial tcp: lookup worldtimeapi.org on 192.168.6.1:53: dial udp 192.168.6.1:53: connect: network is unreachable"

There's another example on node 3407 at 2024-07-16T01:43:13Z.