Log management

EmailEngine logs all its messages to standard output in Pino format. Pino uses JSON structures with some predefined keys, so it is kind of human-readable but not really. Luckily, there are many tools that we can use to transform EmailEngine logs into a more suitable format. The working principle for all these log renderers is the same – you should pipe the standard output from EmailEngine to the log rendering process.

As EmailEngine uses pino logging library, standard pino log levels apply.

• trace (10)
• debug (20)
• info (30)
• warn (40)
• error (50)
• fatal (60)

You can limit output logs by setting a minimum log level. The default is trace, which means that everything is logged.

To change the logging level, either set an environment variable.

EENGINE_LOG_LEVEL=trace

Or use a command-line argument.

$ emailengine --log.level="trace"

jq is a well-known command for processing JSON structures in the command line. It is not EmailEngine specific at all, but we can easily use it to make EmailEngine logs slightly more readable.

$ emailengine | jq

pino-pretty is a command-line utility that makes Pino formatted logs more readable. You can install it from the npm registry as a global command.

$ npm install -g pino-pretty
$ emailengine | pino-pretty

eerawlog is an EmailEngine-specific log renderer. It is meant to help debug IMAP and API call transactions. This is the only thing it's suitable for, so it's not a general log renderer – it tends to throw away everything that is not IMAP-specific. Easier to understand how EmailEngine communicates with IMAP servers. Hard to see how the application is doing in general.

EmailEngine must be started with the --log.raw=true command-line argument or with the EENGINE_LOG_RAW=true environment variable. Otherwise, detailed logs required by the eerawlog command will be disabled.

Detailed logs are disabled by default for the following reasons:

1. raw logs can be pretty large as it includes all bytes sent and received, that includes complete email message sources
2. raw logs do not modify secrets. Regular logs are always processed in a way where account credentials and other secrets are filtered out. Raw logs include all bytes transferred between EmailEngine and the server

$ npm install -g eerawlog
$ emailengine --log.raw=true | eerawlog

pino-gelf is another useful utility that sends all log messages from EmailEngine to a Graylog server. A neat feature is that pino-gelf can also work in a throughput mode so that it both sends the logs to Graylog and logs messages to console so we can mix it up with other tools like pino-pretty

$ npm install -g pino-gelf
$ emailengine | pino-gelf log -h graylog.server.com -t | pino-pretty

Something to keep in mind with pino-gelf is that it creates separate log keys only for root-level log entry keys. Object values are serialized into JSON strings:

{  
  "version":"1.1",
  "host":"emailengine",
  "short_message":"message entry",
  "full_message":"message entry",
  "timestamp":1481840140.708,
  "level":6,
  "_name":"emailengine",
  "_object_value":"{\"key\":\"value\"}",
}

In this case you can't search for object_value.key field in Graylog. Instead there will be a text field object_value with {"key":"value"} as it's value.

If you set EmailEngine up as a SystemsD service, the application logs end up in journald. You can access these logs using the journalctl command.

If you install EmailEngine from DigitalOcean Marketplace or by using the install script, then EmailEngine uses systemd and journald by default.

Tail logs in realtime

$ journalctl -t emailengine -f -o cat | pino-pretty

List logs for the last hour

Journald also allows to retrieve logs with time based queries. The following command lists logs for the last hour.

$ journalctl -t emailengine --since "1 hour ago" -o cat | pino-pretty

Tail prettified logs for a specific account

If you have the eerawlog tool installed and set up, you can filter and pretty print IMAP and API traffic for a specific account on a running system.

$ journalctl -t emailengine -f -o cat | eerawlog --filter.account=account_id