Feature request: seamless compressor migration by deleting old files regardless of old compressor

[arekm]

There is no way to migrate from one compressor to another without having manually recompress old files (as far I know).

This feature request is about adding feature where logrotate will know which extensions
are considered to be still the same log.

Maybe:

compressextlist gz zstd bz2

So it will delete old logs, regardless which compressor was used from the list.

Example of current 3.21.0 behaviour:

hourly
rotate 2
minsize 1k
size 10k
nomail
dateext
dateformat -%Y%m%d%H%M%S
compress
nodelaycompress
missingok
olddir /home/users/arekm/test/logrotate/archive
notifempty
#compresscmd zstd
#uncompresscmd unzstd
#compressext .zst

/home/users/arekm/test/logrotate/log {
}

Logrotate file:

$ logrotate --state /home/users/arekm/test/logrotate/state logrotate.conf; ls -l archive/
total 16
-rw------- 1 arekm users 7382 Jun  8 16:18 log-20230608161842.gz
-rw------- 1 arekm users 7382 Jun  8 16:18 log-20230608161857.gz
$ sudo cat /var/log/messages  > log
$ logrotate --state /home/users/arekm/test/logrotate/state logrotate.conf; ls -l archive/
total 16
-rw------- 1 arekm users 7382 Jun  8 16:18 log-20230608161857.gz
-rw------- 1 arekm users 7382 Jun  8 16:19 log-20230608161904.gz

Now we change from gzip to zstd (uncommenting zstd options from my example config)

$ vim logrotate.conf
$ sudo cat /var/log/messages  > log
$ logrotate --state /home/users/arekm/test/logrotate/state logrotate.conf; ls -l archive/
total 24
-rw------- 1 arekm users 7382 Jun  8 16:18 log-20230608161857.gz
-rw------- 1 arekm users 7382 Jun  8 16:19 log-20230608161904.gz
-rw------- 1 arekm users 7397 Jun  8 16:19 log-20230608161914.zst
$ sudo cat /var/log/messages  > log
$ logrotate --state /home/users/arekm/test/logrotate/state logrotate.conf; ls -l archive/
total 32
-rw------- 1 arekm users 7382 Jun  8 16:18 log-20230608161857.gz
-rw------- 1 arekm users 7382 Jun  8 16:19 log-20230608161904.gz
-rw------- 1 arekm users 7397 Jun  8 16:19 log-20230608161914.zst
-rw------- 1 arekm users 7397 Jun  8 16:19 log-20230608161918.zst
$ sudo cat /var/log/messages  > log
$ logrotate --state /home/users/arekm/test/logrotate/state logrotate.conf; ls -l archive/
total 32
-rw------- 1 arekm users 7382 Jun  8 16:18 log-20230608161857.gz
-rw------- 1 arekm users 7382 Jun  8 16:19 log-20230608161904.gz
-rw------- 1 arekm users 7397 Jun  8 16:19 log-20230608161918.zst
-rw------- 1 arekm users 7397 Jun  8 16:19 log-20230608161920.zst

logrotate never deletes old gzipped logs, only touching zst files.

Proposed feature would make switching between compressor easy, without having to recompress gigabytes of old logs.

[kdudka]

How frequently do you change the compression algorithm?

[arekm]

I think like 3 times so far in my life (gz->bz2->xz->zst) BUT that was only on few of my private machines to keep disk space usage as lowest possible.

On most of machines I still use default gzip since beginning because changing to anything else means recompressing ~36 000 old log files on almost 100 machines (I keep year worth logs; rotated daily) which are about 2TB of data.

If I ever decided to do that, like on my private machines, it would be these numbers x3.

If this feature is problematic in implementation then well... but if it is easy enough then please consider implementing it to make switch easy and convenient for all people like me.

[kdudka]

I think the proposal makes sense but I will defer the final decision to @cgzones and @jamacku who will most likely be reviewing the change in question.