Stale value in 'current-scaling-operation' causing ES Operator to fail

[otrosien]

Expected Behavior

When an index does not exist (anymore) the es-operator should continue to work.

Actual Behavior

When an index that is referenced in the 'current-scaling-operation' doesn't exist anymore, the scaling fails because ES returns a 404 when trying to update number_of_replicas.

es-operator-85d68b858d-q87bh es-operator time="2019-04-28T16:08:43Z" level=info msg="Setting number_of_replicas for index 'index-a' to 1." endpoint="https://es-data-othera.poirot-test.svc.cluster.local.:9200"

apiVersion: zalando.org/v1
kind: ElasticsearchDataSet
metadata:
  annotations:
    es-operator.zalando.org/current-scaling-operation: '{"ScalingDirection":0,"NodeReplicas":3,"IndexReplicas":[{"index":"index-a","pri":5,"rep":1}],"Description":"Keeping
      shard-to-node ratio (35.67), and decreasing index replicas."}'

Steps to Reproduce the Problem

1. scale up data nodes (including increase of index replicas)
2. delete one index allocated on the nodes while the StatefulSet has not yet stabilized
3. check the current-scaling-operation annotation, and watch ES-Operator fail trying to execute the scaling operation.