Kubernetes-Restart support for StatefulSet

[stefanmb]

Feature request

Proposal:

Sieve requested support for restarting StatefulSets with krane, this is due to their use of OnDelete. This request is reasonable and related to #610 and should be implemented with it, but does not necessarily need to be.

From the Sieve team:

It’s not a blocker, just a bit worse than before. Deployments used to take 2 minutes, now they are 46 minutes long. It means the person who wants to ship something will have to wait, multiple commits will be shipped at once, on-call will have to watch them longer if something critical is being rolled out. Plus there are a few cases like crash loops and emergency when on-call will need to manually delete the pods vs just click rollback as before.

@Shopify/app-lifecycle @timothysmith0609 I'm not entirely sure who maintains krane at this point, would anyone on the team have resources to implement this request on behalf of @kgalieva's team?

[kgalieva]

cc @Shopify/change-data-capture

[timothysmith0609]

This looks pretty simple to implement, I'll get on it.

Curiously, we seem to be using a really outdated approach: #61, whereby the pods are triggered to restart by adding a throw-away env var onto the pod itself.

From a verbose krane rollout restart invocation for both deployments and statefulsets, looks like there's a saner, annotation-based, approach:

Request Body: {"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"2021-08-30T15:59:34-04:00"}}}}}
PATCH https://192.168.64.59:8443/apis/apps/v1/namespaces/kube-system/deployments/coredns?fieldManager=kubectl-
rollout 200 OK in 6 milliseconds

Note krane restart already supports extra filtering via label selectors, so it should be pretty straightforward to create separate invocations for deploys, statefulsets, etc, if needed.