High Availability AMQP Messaging with Redundant Queues

Beetle grew out of a project to improve an existing ActiveMQ based messaging infrastructure. It offers the following features:

• High Availability (by using multiple message broker instances)
• Redundancy (by replicating queues)
• Simple client API (by encapsulating the publishing/ deduplication logic)

See RELEASE_NOTES.md

# configure machines

Beetle.config do |config|
  config.servers = "broker1:5672, broker2:5672"
  config.redis_server = "redis1:6379"
end

# instantiate a beetle client

b = Beetle::Client.new

# configure exchanges, queues, bindings, messages and handlers

b.configure do
  queue :test
  message :test
  handler(:test) { |message| puts message.data }
end

b.publish :test, "I'm a test message"

b.listen_queues

Beetle ships with a number of example scripts.

The top level Rakefile comes with targets to start several RabbitMQ and redis instances locally. Make sure the corresponding binaries are in your search path. Open four new shell windows and execute the following commands:

rake rabbit:start1
rake rabbit:start2
rake redis:start:master
rake redis:start:slave

To set up a redundant messaging system you will need

• at least 2 AMQP servers (we use RabbitMQ)
• at least one Redis server (better are two in a master/slave setup, see REDIS_AUTO_FAILOVER.md)

For testing purposes, you will need a MySQL database with the database beetle_test created. This is needed to test special cases in which Beetle handles the connection with ActiveRecord:

 mysql -e 'create database beetle_test;'

You also need a Redis instance running. The default configuration of Redis will work:

redis-server

If you want to run the integration tests you need GO installed and you will need to build the beetle binary. We provide a Makefile for this purpose, so simply running

make

should suffice.

At runtime, Beetle will use

• bunny
• redis
• amqp (which is based on eventmachine)
• daemons
• activesupport

For development, you'll need

• mocha
• cucumber
• daemon_controller
• consul

For tests, you'll need

• activerecord
• mysql2

Dependencies are managed by bundler.

If you want to use a redis-rb version after 5.0.0, you must add the hiredis-client gem to your application.

Stefan Kaes, Pascal Friederich, Ali Jelveh, Bjoern Rochel and Sebastian Roebke.

You can find out more about our work on our dev blog.

Copyright (c) 2010-2019 XING AG

Released under the MIT license. For full details see MIT-LICENSE included in this distribution.

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Hack along and test your code.
4. Commit your changes (git commit -am 'Add some feature')
5. Push to the branch (git push origin my-new-feature)
6. Create new Pull Request

Don't increase the gem version in your pull requests. It will be done after merging the request, to allow merging of pull requests in a flexible order.

In order to execute the unit tests, you need Ruby, a running rabbitmq server, a running redis-server, a running mysql server and a runnning consul server.

In addition, beetle ships with a cucumber feature to test the automatic redis failover as an integration test. For this you need a recent Go installation in order to compile the beetle go binary. Just invoke make in the top level directory.

There are two ways to start the required test dependencies: using docker-compose or starting the services manually.

Open a separate terminal window and run

 docker-compose pull

followed by

 docker-compose up

This will start mysql, two redis servers, two RabbitMQ instances and a single consul development node.

Note: make sure to wait until all services are properly started.

The top level Rakefile comes with targets to start several RabbitMQ instances locally. Make sure the corresponding binaries are in your search path. Open three shell windows and execute the following command:

rake rabbit:start1

and

rake redis:start:master

as well as

rake consul:start

Then you can run the cucumber feature by running:

cucumber

or

rake cucumber

Note: Cucumber will automatically run after the unit tests when you run rake without arguments.

Update RELEASE_NOTES.md!

We use semantic versioning and create a git tag for each release.

Edit lib/beetle/version.rb and go/src/github.com/xing/beetle/version.go to set the new version number (Major.Minor.Patch).

In short (see semver.org for details):

• Major version MUST be incremented if any backwards incompatible changes are introduced to the public API.
• Minor version MUST be incremented if new, backwards compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated.
• Patch version MUST be incremented if only backwards compatible bug fixes are introduced.

Then use rake release which will create the git tag and upload the gem to github.com:

bundle exec rake release

The generated gem is located in the pkg/ directory.

In order to build go binaries and upload the docker container with the beetle GO binary to docker hub, run

make release

This will upload the go binaries to https://github.com/xing/beetle/ and push the beetle container to https://hub.docker.com/r/xingarchitects/gobeetle/.

Run

make tag push TAG=X.X.X

to tag and push the container with a specific version number.