Add compute node log forwarding

[kkm000]

Carryover from #17

Currently, in our private setup, a manually added hack, a node shutdown script essentially saves journalctl -b | gzip -c > uniquefile on a node shutdown. It's an unsustainable solution. Proper log forwarding is essential.

Fluentd is a monster, Fluent Bit looks much more reasonable.

[jtrmal]

I think even several "normal" syslogs can direct the outputs to a remote machine y.

…

On Thu, Dec 10, 2020 at 11:31 PM kkm000 ***@***.***> wrote: Carryover from #17 <#17> Currently, in our private setup, a manually added hack, a node shutdown script essentially saves journalctl -b | gzip -c > uniquefile on a node shutdown. It's an unsustainable solution. Proper log forwarding is essential. Fluentd is a monster, Fluent Bit looks much more reasonable. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUKYXZG67NNSMSQRZTEAV3SUFD2LANCNFSM4UVWJVSQ> .

[kkm000]

Yeah, there is a "classic" (RFC number < 1000) UDP syslog protocol, and journald can forward, too, even more efficiently (batching, compression). Journal is much better than syslog, as it records a lot of metadata with the event, and journald both listens on internal Unix syslog socket and can be a syslog server. Nothing wrong in principle with forwarding to e.g. the login node's. The GCP logging is a bit preferable, however, because it is possible to create monitoring and alerting metrics out of arbitrary log queries, kinda what Splunk and other monitoring software does. Fluent Bit supports GCP logging natively, and is very compact. Own GCP solution is based on Fluentd, which is made by the same company, but much more complex, uses significantly more CPU and memory, and is capable of much more that we need. Fluent Bit is a single C++ binary that can start even before / is remounted rw, as soon as journald starts, if given temp space in /run. I'm surprised they preferred Fluentd for simple flat host-to-cloud logging.