Add the TCP sink. Add support for log group name and log stream name. Add the Units reference.

[nscott]

Description

This adds a TCP and Async sink. These allow non-Lambda deployments (e.g. ECS Fargate) to communicate to a CloudWatch Log Agent running as a side-car.

The Async sink can wrap any sink to allow processing to happen in the background. This ensures that production code isn't affected by network communicate to the agent.

There's also a new Units file that references all of the known units for CloudWatch.

Changes

The TCP sink will attempt to connect to the TCP endpoint up to 3 times before giving up. Errors will be sent back over a logger if it was supplied.

The Async sink will start a new thread when it's created, wrapping an existing sink. #accept pushes messages to a queue, which is #pop'd by the thread and sent to the sink's #accept.

Messages that exceed the maximum queue size are dropped.

Updated Dependencies

• I added the tcp-client gem, v0.11.2. I originally wrote my own client, but later thought better of that.
• Concurrent-ruby automatically upgraded itself from 1.1.7 to 1.1.10.

Ticket

N/A

Screenshots

N/A

Notes

Recommended reading: Code Review guide

Optional Tasks

• Include 🎩 Instructions
• Update the readme (README.md)
• Update the API or architecture docs (e.g. docs/api.md)

Library-Specific

• Increment the changelog (CHANGELOG.md)
• Increment the version number (lib/version.rb)
• Release & Tag the version above in Github

Performance

It's working in a beta environment serving real traffic and performance seems stable :)

[metaskills]

Wow, nice work! LGTM. Do you feel this is ready to merge?

[nscott]

Thanks!

Over the weekend I was thinking it was ready, but after letting the Async version bake in our beta environment I'm not so sure. We use Rails+Puma.

I was reading about Puma, and based on my config we only have 5 threads running. I expected 10 threads to be running based on where I put the async initializer. We don't use workers, so just 1 process. NewRelic didn't show more threads being used which was concerning. We DID see CPU spikes cycling with very little traffic, which was unexpected. I dropped to just the plain TCP client and everything stabilized.

I was reading about Async in Ruby 3 and I think that's a better approach, but I also think the threaded approach could be useful for folks that have different thread/process model.

Do you think I should rip out the threaded version and/or rename it? I noticed there's also at least 1 threading bug with regards to shutdown, although in practice I think it's not a problem. You can add messages to a queue based on arbitrary interleaving once you call shutdown since it adds the StopMessage to ensure pop can be called once more to allow the thread to exit.

[metaskills]

Initial comments. See peer dep ideas.

[metaskills]

Re async stuff. Not sure. Are there patterns from other gems to take hints from? For example, I know a lot of gems like Rollbar which have async handlers that would use SuckerPunch for backend. Is there an opportunity here to do something similar? cc @drinks

[nscott]

Hey, so sorry to let this languish. I'll get back to this shortly. I've decided to remove Async pieces. I'm still looking into the peer dependency piece for tcp-client.

[nscott]

@metaskills Whew, finally got around to making those changes. Bit shameful, but done! The Async sink is removed, the docs are updated, and tests pass.

I've been running the Tcp sink in my production service for a few months and it's looking 💯

[nscott]

@metaskills Just checking in on this PR :) I know I let it linger a long time. Any chance you could take a look and potentially merge it? Thanks!

[metaskills]

DONE! Thanks for the reminder. Published v.0.8.0 to ruby gems.