sshd jail fails on Ubuntu 20.04 (is this module maintained?)

[tobixen]

Rolling out the module according to the README on Ubuntu 20.04 just for protecting sshd causes a broken fail2ban ... the service fails with the error message:

fail2ban                [878455]: ERROR   Failed during configuration: Have not found any log file for sshd jail

I can probably figure out of this and make a pull request, but first I'd like to know if this module is maintained, and if it is supposed to work with Ubuntu? The alternative seems to be the voxpupuli-module, but it seems like it's a lot more work to set up a simple sshd jail with the voxpupuli-module.

[tobixen]

Looking more into the fail2ban configuration, the sshd_backend is configured to be a default_backend which again is defined to be default/backend - and then I'm not able to figure out where default/backend is configured. If I replace this with systemd, fail2ban starts working.

Apparently the default backend is not touched by this fail2ban puppet module (and the default sshd configuration in the package is the same as what was rolled out by this fail2ban-module). Perhaps the problem is actually in the fail2ban package, and not in the puppet module. I will look a bit more into it.

[tobixen]

I purged the config, reinstalled the package, and enabled the sshd jail according to the instructions supplied in the jail.conf file from the package - and fail2ban fails to start. I conclude that the problem is in the fail2ban package for ubuntu, not in this puppet module as such. Probably it will work if installing a syslog package.

I will leave this issue open - I think that with a good puppet module things should "just work" - even if that means adding workarounds for known bugs in the package. :-)

[tobixen]

Found some relevant issues:

https://bugs.launchpad.net/ubuntu/+source/fail2ban/+bug/1696591
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770171

So in the default /etc/fail2ban/jail.conf the default backend is set to systemd. Since jail.conf is maintained by this puppet module, I've concluded that the issue should be addressed by this puppet module, regardless of how it's solved upstream.

Setting the default backend to systemd may not be the correct approach, as it will break for services that uses log files without explicitly giving a backend.

[lelutin]

Hi and thanks for reporting the issues with ubuntu 20.04!

and yes this module is still being maintained. I still use it in production, but on debian -- I don't tend to use Ubuntu so I can't easily spot those problems. So I need help with making the module work for ubuntu and I'd be happy to merge in changes to fix things up for it.

aha, so there's no syslog log file in ubuntu 20.04?

what we can do then since ubuntu, seems to require a different set of options, would be to create a new file in data/Ubuntu.yaml (guessing the major version exposed by facts.os.distro.release.major, it might be different than "20") to add an override to the backend parameter.
We can either override just the sshd jail (for that we need to copy the parameters from common.yaml and add the backend key to it -- you can see an example of overrides in the Debian.yaml file) or go with the global fail2ban::backend param if that seems more relevant -- but in that case we'd need to make sure that all the other jails are still working, or override the backend param for those that need to still get info from a log file.

Do you know if the same problem would happen in 22.04 also?

[tobixen]

Syslog can be installed as a separate package. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770171 hints that this may be an issue on Debian too ... all since 2014 indeed :-) The bug report I found for Ubuntu tells me it has been an issue also for ubuntu 16.04 (but we do have several ubuntu 16.04-instances with your puppet module running where it hasn't been an issue ... perhaps our 16.04 base image came with syslog installed, I should check that up). This will definitively continue being an issue in 22.04, unless the problem is fixed upstream.

I'll think a bit more on this until tomorrow.

[tobixen]

I should add, this (log information from ssh and others only available in the systemd journal) is not specific for Ubuntu, this is the general directions things are going in most distributions using systemd. I'll check up a bit how things are working on centos and archlinux as soon as I get the time for it.

[tobixen]

I installed fail2ban "out of the box" on archlinux, enabled the sshd jail as suggested in jails.conf, and ... it works. The reason is the file paths-arch.conf that looks like this:

# Arch

[INCLUDES]

before = paths-common.conf

after  = paths-overrides.local


[DEFAULT]

apache_error_log = /var/log/httpd/*error_log

apache_access_log = /var/log/httpd/*access_log

exim_main_log = /var/log/exim/main.log

mysql_log = /var/log/mariadb/mariadb.log
            /var/log/mysqld.log

roundcube_errors_log = /var/log/roundcubemail/errors

# These services will log to the journal via syslog, so use the journal by
# default.
syslog_backend = systemd
sshd_backend = systemd
dropbear_backend = systemd
proftpd_backend = systemd
pureftpd_backend = systemd
wuftpd_backend = systemd
postfix_backend = systemd
dovecot_backend = systemd

There is also a paths-fedora.conf that looks quite similar.

I think that for all modern systemd-based distros, the logging for those services will be available in the systemd journal, and the logging may also be available in legacy logs if the syslog or rsyslog package is installed and the service is running. This may be optional on some OS'es ... like, as referenced, someone already reported in 2014 that they had a Debian-system running without syslog.

I have checked up some systems:

• My archlinux laptop does not have any syslog package installed, and the logging from sshd etc is exclusively found in the journal.
• Same for a centos8 instance and some Ubuntu 20.04-instances.
• Our instances running centos7 and Ubuntu 16.04 all have rsyslog installed and logs available.

My suggestion is to roll out a file /etc/fail2ban/paths-systemd.conf containing this:

syslog_backend = systemd
sshd_backend = systemd
dropbear_backend = systemd
proftpd_backend = systemd
pureftpd_backend = systemd
wuftpd_backend = systemd
postfix_backend = systemd
dovecot_backend = systemd

... and make sure it's included in all modern systemd-based distros.

[tobixen]

I've raised an issue upstream: fail2ban/fail2ban#3292

[tobixen]

@lelutin , could you check if my hypotheses are correct also for your Debian ...

• It does use systemd
• information from ssh, postfix, proftpd, wuftp, dropbear, etc does end up in the journal?

I don't think it's a point to check all those services ... just do sudo journalctl -f and try to ssh into it.

[lelutin]

Hi again @tobixen and thanks for all of the analysis.

So far the approach that I've used for this puppet module is to make sure that default configurations in distros/releases are supported out of the box by the module, and it should be possible to set options to make it configure things when a different setup is used.

So.. if I understand the situation properly, do you mean that ubuntu's default configuration (e.g. a fresh vanilla install of ubuntu) replaced syslog with journald entirely starting with a certain version (maybe 20.04? or maybe before?). I know that debian has still not moved to replace syslog with journald, but it's quite possible that ubuntu made the move.
If that's the case, then we can add some overrides in this module in order to accomodate this change.

One method is to override the default backend param. The module has a backend param on the main class that sets the default backend. So we could override that parameter via hiera where necessary -- e.g. we could have a default case in Ubuntu.yaml that would reflect the new configuration of using systemd as the backend, and then add Ubuntu-18.yaml with an override to set the bakend back to auto and then copy this to other Ubuntu-X.yaml files for the other supported versions of ubuntu that still were using syslog by default.

Do you think this method would work sufficiently well for you and other users? If you want to try it out, you can create a file data/Ubuntu-20.yaml in the module on your puppet server with content like this:

---
fail2ban::backend: 'systemd'

This'll set the backend only for 20.04. If this works properly we can create the files in the module in the "reverse order" as described above (e.g. create the above content in data/Ubuntu.yaml instead, and then add similar content for data/Ubuntu-18.yaml and data/Ubuntu-16.yaml but with auto instead of systemd.

Also, by looking at things for this issue, I saw that management of paths-* should have been removed with removal of debian jessie support, so I created #57 to sidestep that issue in its own place.

[lelutin]

oh wait I think I got confused a bit (it's a bit hard for me to visualize what's happening without a local ubuntu VM -- I should just download an image to check things out).

iiuc the current value within the paths-* file is something like:

default_backend = %(default/backend)s

sshd_backend = %(default_backend)s

is this it?

if that's the case then I believe it means that with %(default/backend)s it is grabbing the default backend set in jail.conf in the default section. So in that case the difference that would be an issue for your case would be the value in jail.conf and setting fail2ban::backend to the value systemd should make your setup work properly (e.g. there's no weirdness happening with management of the paths-* files since the module is apparently not managing them at all)

Sorry if I sounded maybe a bit confused in my previous message. But as I said in the previous message, if ubuntu did change the default backend for logs from syslog to journald starting with 20.04 we should apply a fix with hiera as described previously.

Cheers

[tobixen]

Unfortunately the fail2ban setup got pushed further down in my todo-stack, I'm not sure I will get time to revisit this one until the autumn. If I remember correct, the ubuntu image I had would only log things like sshd into the journal and not into log files. I do not know if this is specific for the ubuntu image I had access to or if it applies generally. As far as I can understand, Debian can also be configured that way, as I found an old bug report on this applying to Debian.