This script allows handling of S3 logs as incoming streams to be either piped and colored on the fly from the console, or to consolidate them into a file
It resorts to long polling to update the log as AWS doesn't ship in real time.
It's designed to work with logs produced by s3-streamlogger and winston.
You need to have an AWS S3 bucket and
it's recommended to set a new AWS IAM user with these permissions:
then store his access keys in your ~/.aws/credentials.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": ["arn:aws:s3:::muzetestlogs",
"arn:aws:s3:::your-bucket-name/*"]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": ["arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"]
}
]
}npm install -g s3logtoolsEnsure that the AWS credentials to access your bucket are set in your
environment
(for example by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
environment variables) or in the ~/.aws/credentials.
s3logtools --bucket=your-log-bucket-name --watchMore complex operations are possible:
s3logtools --bucket=your-log-bucket-name --ago='30 days' --output='myfile.csv' --delimiter='|'Full list:
| Options | Use |
|---|---|
| -v | prints the version number |
| -o, --output <path/filename> | compacts the response into a correctly formatted file (.json/.csv) |
| -w, --watch | 'Live tails using polling' |
| -s, --start | starts the logging output from this time (momentjs compatible)') |
| -e, --end | ends the logging output after this time (momentjs compatible)') |
| -a, --ago | starts the logging output time ago (momentjs compatible)') |
| --prefix | fetch only logs with the specified prefix') |
| -b, --beautify | beautify the output') |
| -r, --raw | outputs the logs as is from s3') |
| -p, --profile | AWS profile name to use from ~/.aws/credentials') |
| -f, --filter | filters only the logs matching the specified pattern') |
| -l, --loglevel | what level of logs to report: silent, error, minimal, warn, info, verbose, silly (default: warn) |
ISC: equivalent to 2-clause BSD.