This is a AWS Kinesis input plugin for Logstash. Under the hood uses the Kinesis Client Library.

This plugin requires Logstash >= 1.5, and can be installed by Logstash itself.

$(LOGSTASH_DIR)/bin/plugin install logstash-input-kinesis

input {
  kinesis {
    kinesis_stream_name => "my-logging-stream"
    codec => json { }
  }
}

This are the properties you can configure and what are the default values:

• application_name: The name of the application used in DynamoDB for coordination. Only one worker per unique stream partition and application will be actively consuming messages.
  • required: false
  • default value: logstash
• kinesis_stream_name: The Kinesis stream name.
  • required: true
• region: The AWS region name for Kinesis, DynamoDB and Cloudwatch (if enabled)
  • required: false
  • default value: us-east-1
• checkpoint_interval_seconds: How many seconds between worker checkpoints to DynamoDB. A low value ussually means lower message replay in case of node failure/restart but it increases CPU+network ussage (which increases the AWS costs).
  • required: false
  • default value: 60
• metrics: Worker metric tracking. By default this is disabled, set it to "cloudwatch" to enable the cloudwatch integration in the Kinesis Client Library.
  • required: false
  • default value: nil

This plugin uses the default AWS SDK auth chain, DefaultAWSCredentialsProviderChain, to determine which credentials the client will use.

The default chain follows this order trying to read the credentials:

• AWS_ACCESS_KEY_ID / AWS_SECRET_KEY environment variables
• ~/.aws/credentials credentials file
• EC2 instance profile

The credentials will need access to the following services:

• AWS Kinesis
• AWS DynamoDB: the client library stores information for worker coordination in DynamoDB (offsets and active worker per partition)
• AWS CloudWatch: if the metrics are enabled the credentials need CloudWatch update permisions granted.

Look at the documentation for deeper information.

1. Fork it ( https://github.com/codekitchen/logstash-input-kinesis/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request