A Node.js AWS Lambda script that send S3 logs to Loggly
Note: This script has been deprecated in favor of Loggly's new built-in S3 ingestion feature. You can learn more in the S3 ingestion docs.
- Clone the git repo
git clone https://github.com/psquickitjayant/S3ToLoggly.git
cd S3ToLoggly
- Install require npm packages.
npm install
- zip up your code
zip -r S3ToLoggly.zip S3ToLoggly.js node_modules
The resulting zip (S3ToLoggly.zip) is what you will upload to AWS in step 1 below.
For all of the AWS setup, I used the AWS console following this example. Below, you will find a high-level description of how to do this. I also found this blog post on how to set things up using the command line tools.
- Create Role
- Sign in to your AWS account and open IAM console https://console.aws.amazon.com/iam/
- In your IAM console create a new Role say, 's3-execution-role'
- Select Role Type as 'AWS Lambda'
- Apply policy 'AmazonS3FullAccess' and save.
- Create lambda function
- https://console.aws.amazon.com/lambda/home
- Click "Create a Lambda function" button. (Choose "Upload a .ZIP file")
* Name: S3ToLoggly
* Upload lambda function (zip file you made above.)
* Handler:* S3ToLoggly.handler
* Role:* In the drop down click "s3-execution-role". (This will open a new window to create the role.)
* I left the memory at 128MB. In my testing with s3 bucket set upload every 5 minutes this worked for me. You may need to bump this up if your s3 logs are larger.
* Same advice for Timer, I set it to 10 seconds. - Configure Event Source to call S3ToLoggly when logs added to S3 bucket.
- https://console.aws.amazon.com/lambda/home
- Make sure the S3ToLoggly lambda function is selected, then click 'Actions->Add event source' * Event source type: S3 * Bucket: Choose the S3 bucket that contains your logs. * Event type: ObjectCreated (All)
### Configure the S3 buckets with tags the S3ToLoggly uses to know where to send logs.
Using S3 Management Console click the bucket that contains your S3 logs.
- Under Properties -> Tags add the following tag:
1. **Key:** loggly-customer-token , **Value:** *your-loggly-customer-token*
2. **Key:** loggly-tag, **Value:** *s3_lambda_tag* (Or what ever you want.)
NOTE: This script sends raw logs to Loggly. If you want to parse your own logs and convert them to JSON, you can extend the functionality of exports.handler
function. JSON logs can be automatically parsed in Loggly. See more for JSON Automated Parsing in Loggly.