HostnameGenerator is flaky when there are collisions

[lobkovilya]

What happened?

1. Force HostnameGenerator to have collisions
2. Observe the hostname in meshservice.status is constantly changing from the real hostname to the error message

We noticed this in the multizone scenario with statefulsets, but the repro could be way simpler:

1. Apply statefulset in 2 zones:

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: redis
     namespace: kuma-demo
   spec:
     selector:
       matchLabels:
         app: redis
     serviceName: redis
     replicas: 0
     template:
       metadata:
         labels:
           app: redis
       spec:
         containers:
           - name: redis
             image: "redis"
             ports:
               - name: tcp
                 containerPort: 6379
             env:
               - name: POD_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.name
             lifecycle:
               preStop: # delay shutdown to support graceful mesh leave
                 exec:
                   command: ["/bin/sleep", "30"]
               postStart:
                 exec:
                   command: ["sh", "-c", "/usr/local/bin/redis-cli set zone ${POD_NAME}"]

2. In one zone apply the hostname generator (without kuma.io/origin label):

   kind: HostnameGenerator
   apiVersion: kuma.io/v1alpha1
   metadata:
     name: local-headless-kube-mesh-service
     namespace: kuma-system
     labels:
       kuma.io/origin: zone
   spec:
     template: '{{ label "statefulset.kubernetes.io/pod-name" }}.{{ label "k8s.kuma.io/service-name" }}.{{ .Namespace }}.svc.mesh.local'
     selector:
       meshService:
         matchLabels:
           kuma.io/managed-by: k8s-controller
           k8s.kuma.io/is-headless-service: "true"

3. Observe hostnames appearing and disappearing
   
4. Zone CP is constantly reconciling resources zone-cp.log