skip_index parameter to support regular expressions

[EduSandu]

Taking the following scenario into discussion where we would have the ES cleaner run against all indices in an ES cluster with the exception of a particular indice pattern, for example to exclude all indices matching openshift-infra-%Y.%m.%d.

I've tested adding an expression as openshift-infra-* to the skip_index parameter and when invoked manually, it finds the respective indice but also returns:

• "Keeping index: openshift-infra-2019.10.01"

For the default .kibana,.kibana_5 indice that "Keeping..." output is not returned, thus I assume it is skipped successfully.

Not very familiar with Python, thus I want to ask if the skip_index variable supports delimiting indices through regular expressions? Alternatively I guess the indices can be defined in the index variable instead of all value, which also raises the same question, if it supports expressions? :)

[giuliocalzolari]

@EduSandu that sounds like a great idea.
a similar issue was resolved in #26 would be awesome to have some help from the community to implements regular expression for the index to skip and the index to delete.

I will try a beta version in a dedicate branch

[giuliocalzolari]

@EduSandu please check the new version
https://github.com/cloudreach/aws-lambda-es-cleanup/blob/e0f600561e2f0914befb2f8c236658ff47cfe8f0/es-cleanup.py

and test it.

[EduSandu]

@giuliocalzolari awesome stuff - tested OK it matched and processed all indexes using the wildcard match and also skipped the .kibana indexes.

I omitted initially during testing to change the skip_index param, which was using the comma-delimited values - the cleaner job deleted the .kibana_N indexes, but was able to restore them from a snapshot.

Wondering if the .split(",") delimeter can be kept and used to separate multiple expressions in the index & skip_index variables?
This gives more granular control and lets you to keep more important indices longer (have them skipped via the expression matching their pattern in skip_index) and drop the ones that are not that relevant earlier, in order to save disk usage.

[giuliocalzolari]

@EduSandu uhmm very wierd...
I've setup the skip_index to also use the regexp

aws-lambda-es-cleanup/es-cleanup.py

Line 56 in e0f6005

self.cfg["skip_index"] = self.get_parameter("skip_index", ".kibana*")

are you using Terraform to deploy it?
because I've realised the default value for terraform is still ".kibana" and this pattern does not match .kibana_N

can you confirm it?

[EduSandu]

@giuliocalzolari no this wasn't a new domain standed up using Terraform.
skip_index works using the regexp method, only the initial test cleared the .kibana_N indices (skip_index was still using the prior comma delimited format)

Would like to test having multiple expressions with a , delimeter, for the reasons mentioned above - guessing the following lines

aws-lambda-es-cleanup/es-cleanup.py

Lines 55 to 56 in e0f6005

	self.cfg["index"] = self.get_parameter("index", ".*")
	self.cfg["skip_index"] = self.get_parameter("skip_index", ".kibana*")

should be amended as such? Not sure if re.search will support it
self.cfg["skip_index"] = self.get_parameter("skip_index", ".kibana*").split(",")

[giuliocalzolari]

The Regexp done like that

 self.cfg["skip_index"] = self.get_parameter("skip_index", ".kibana*") 

can match multiple patterns/indexes
I will open a new PR with the current changes

@EduSandu thanks for your support!

[giuliocalzolari]

#33