Feature request: treat multiple files as group

[wyohm]

As mentioned in Add a "minage" feature #28, some services like Tomcat write their logs with a date in the filename, which doesn't work so well with logrotate. "minage" is a good feature, but it limits functionality (e.g. compress) since it's only very useful with "rotate 0", otherwise it will create identical files like

localhost_access_log.2018-01-17.txt
localhost_access_log.2018-01-17.txt.1
localhost_access_log.2018-01-17.txt.2
localhost_access_log.2018-01-18.txt
localhost_access_log.2018-01-18.txt.1 
localhost_access_log.2018-01-18.txt.2
localhost_access_log.2018-01-19.txt
localhost_access_log.2018-01-19.txt.1
localhost_access_log.2018-01-19.txt.2

There are also other examples of people having difficulty solving this problem with logrotate. The answers are typically, "use a cron job rather than logrotate", or "have logrotate run a script". I'd really like to be able to keep all my log rotation and deletion policies within logrotate.
It's possible to use a 'postrotate' script like:

'find -iname "localhost_access_log.\*.txt" -mtime +1 -exec xz {} \;'

but if the filename were like 'localhost_access_log.2018-01-17', then "localhost_access_log.*" would also match the compressed files so the 'find' becomes more complicated.

I have a couple of ideas to cover this case:
1:
'compressminage' directive:

/var/log/tomcat/localhost_access_log.*.txt {
	daily
	compress
	rotate 0
	minage 7
	compressminage 1
	}

With the above (proposed) setting, I'd expect matching files older than 1 day to be compressed, and then removed after 7 days. This is a little awkward since "compressminage" is less than the "minage".

2:
A 'collate' directive. This would cause the block to act on any matched files as a group.

/var/log/tomcat/localhost_access_log.*.txt {
	daily
	collate
	compress
	delaycompress
	rotate 7
	}

With the above, I'd expect the youngest log file to be left alone, the rest to be compressed, and at most 7 compressed log files to be retained. This looks like a bigger development effort to me, but is nice since it's more understandable and has less interdependence between the directives (in suggestion 1, changing daily to weekly would also require changing minage to 49 in order to keep the last 7 logs). Possibly, the 'rotate' directive would be replaced with 'retain' in order to make clear that there is no actual rotation happening (just compression and removal).

3:
'norotate' directive and modify 'maxage' and behavior:

/var/log/tomcat/localhost_access_log.*.txt {
	daily
	norotate
	compress
	delaycompress
	maxage 7
	}

I'd expect the youngest log file to be left alone, the rest to be compressed, and matching files (and implicitly the same pattern with the 'compressext') to be removed after 7 days. This has the advantage of being entirely date based (can also mean stateless) and I'd estimate it to be relatively easy to implement. On the downside, it also requires a new directive, and changing the behavior of 'maxage' to operate even if there will be no rotation.

Regardless of which way it is accomplished, can we cover this use case of logs with dates in their names?
Thanks!

[kdudka]

I think the root cause is that the service has its own log rotating facility, which fights with logrotate. Could this be solved by configuring the service such that it writes the log to a single file named localhost_access_log.txt and let logrotate create all the other files with timestamp suffixes?

[wyohm]

Agreed, that's the root cause. And changing it so it writes a single log localhost_access_log.txt probably is the best option, and I try to steer things in this direction. But this isn't always possible. I gave Tomcat as the most common example I'm aware of, but some are not easy to change (proprietary systems), or other team members don't want to change. And some services just require it: atop seems to be hard coded to use its date-named format.

[mat-m]

@wyohm : shouldn't this work:

/var/log/tomcat/localhost_access_log.*.txt {
	daily
	rotate 1
	compress
	delaycompress
	maxage 7
        minage 1
	}

My understanding is: Each day, it rotates and compresses files older than one day, and remove rotated (and compressed) files older than 7 days.

[wyohm]

@mat-m I haven't worked with this recently, but I think that will create the problem I brought up about minage: creating identical files. So if you have:

localhost_access_log.2018-01-17.txt
localhost_access_log.2018-01-18.txt
localhost_access_log.2018-01-19.txt

Logrotate with those settings will produce something like:

localhost_access_log.2018-01-17.txt
localhost_access_log.2018-01-17.txt.1
localhost_access_log.2018-01-17.txt.2
localhost_access_log.2018-01-18.txt
localhost_access_log.2018-01-18.txt.1 
localhost_access_log.2018-01-18.txt.2
localhost_access_log.2018-01-19.txt
localhost_access_log.2018-01-19.txt.1
localhost_access_log.2018-01-19.txt.2

But I haven't tested this and it's been a while since I wrote Logrotate directives.