This script allows handling of S3 logs as incoming streams to be either piped and colored on the fly from the console, or to consolidate them into a file
It resorts to long polling to update the log as AWS doesn't ship in real time.
It's designed to work with logs produced by s3-streamlogger and winston.
You need to have an AWS S3 bucket and
it's recommended to set a new AWS IAM user with these permissions:
then store his access keys in your ~/.aws/credentials
.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": ["arn:aws:s3:::muzetestlogs",
"arn:aws:s3:::your-bucket-name/*"]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": ["arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"]
}
]
}
npm install -g s3logtools
Ensure that the AWS credentials to access your bucket are set in your
environment
(for example by setting the AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables) or in the ~/.aws/credentials
.
s3logtools --bucket=your-log-bucket-name --watch
More complex operations are possible:
s3logtools --bucket=your-log-bucket-name --ago='30 days' --output='myfile.csv' --delimiter='|'
Full list:
Options | Use |
---|---|
-v | prints the version number |
-o, --output <path/filename> | compacts the response into a correctly formatted file (.json/.csv) |
-w, --watch | 'Live tails using polling' |
-s, --start | starts the logging output from this time (momentjs compatible)') |
-e, --end | ends the logging output after this time (momentjs compatible)') |
-a, --ago | starts the logging output time ago (momentjs compatible)') |
--prefix | fetch only logs with the specified prefix') |
-b, --beautify | beautify the output') |
-r, --raw | outputs the logs as is from s3') |
-p, --profile | AWS profile name to use from ~/.aws/credentials') |
-f, --filter | filters only the logs matching the specified pattern') |
-l, --loglevel | what level of logs to report: silent, error, minimal, warn, info, verbose, silly (default: warn) |
ISC: equivalent to 2-clause BSD.