Admin-level audit logging

[wohali]

The idea is to log, with detail, any admin-level changes made thru the HTTP interface (or Erlang remsh? Hm.)

Logging any configuration changes to a node for posterity is useful for documentation and forensic purposes. At a minimum, I propose to force-log the entire payload to any /_node/*/_config endpoint, regardless of logging level.

This could be disable-able, but that would defeat the purpose, since a hacker finding a way around in-built security features would simply reconfigure that value first, before proceeding to more sensitive endpoints. So, perhaps, it's an opt-in thing you enable from vm.args, with the understanding that secrets may be logged in it (e.g. changing/setting a password).

Context

Forensic investigation of hacked CouchDB servers (due to disclosed CVEs) is complicated by the fact that logging of what config changes have been made is terse to the point of uselessness, even at debug level.

[wohali]

This could be opt-in through a vm.args thing, perhaps, to prevent being disabled at run-time.

And it could possibly log to a different file.

Updating the description to be clearer about the intent.

[rnewson]

Is it assumed that an attacker (a successful one) would be unable to load their own version of whatever code module this forcing is in?

[nickva]

I can see the HTTP interface changes but once they have access to remsh it's pretty much all over. Logs would have to be shipped off to a network location to trace any steps taken before they got control of the machine.

Though there might be another usage of this - auditing changes made by customers to help support debug any issues later on. That is the scenario where a customer set some parameters and their cluster started misbehaving. They call support and say "CouchDB stopped working". It might be useful to examine what changes they made and when.

[wohali]

@nickva that's the primary use case. Dealing with forensics is a secondary use case - if they're smart enough to hack intelligently, they'll also be smart enough to clean up after themselves. Then again, maybe not.

[wohali]

@rnewson I mean cryptographically signing Erlang modules and ensuring the exact correct revision / reproducible binaries are used would be lovely, but that's outside the scope of this enh request.

[rnewson]

If there's a vm.args level flag for this I'm fine with the intention.

remsh can be protected with a restricted shell (http:https://erlang.org/doc/man/shell.html).

[iilyak]

At a minimum, I propose to force-log the entire payload to any /_node/*/_config endpoint, regardless of logging level.

I think we should make an exception about entire payload for admins section.

[wohali]

@iilyak Maybe 2 logging levels - one is absolutely everything, and one is "everything but passwords?" I definitely see a use case for logging everything.

[iilyak]

I definitely see a use case for logging everything.

@wohali: What is the use case when knowing user's password is needed, I am curious?

If it is user forget their password or have typo it should be solved through other means IMO. Otherwise there is no protection from curious operator. If password is known then normal audit trail wouldn't help. Because requests from user and curious operator are not distinguishable (when they access via tor or something).

[wohali]

@iilyak Hacker scenarios.

[janl]

+1

[mwasson74]

+1 in 2024