Chore is a system health monitoring tool for everyone.
There are more sophisticated tools, tools with better disaster recovery, tools that monitor CPU/Swap/Disk-space, to be sure. But these usually require switching gears from your main codebase, an advanced learning curve, and often find themselves used only by a dedicated systems person or a single resident expert on your team.
Chore provides a simple API that anyone can start using in 5 minutes or less. This allows any developer to add their own monitoring to tasks with little to no effort.
Chore provides a simple ruby interface to indicate:
-
When a task is started. (Is it actually running?)
-
Optionally when it finished and/or errored. (Is it completing?)
-
Optionally parameters such as how frequently the job should run and/or how long it should take. (Is it behaving as expected?)
Chore does not:
-
Perform automated recovery of failed tasks.
-
Kill or restart zombie processes.
-
Provide server health info such as high CPU usage, low disk space, excessive swapping, etc.
To start a server:
chore-server
This will start a server.
To view status from the command line:
chore-status [hostname] [port]
Which will produce output like:
crazy_background_task - started 2012-06-11 17:11:16 -0400 (Job should run every 1 minute, but has a grace period of 40 minutes)
random_resque_job - started 2012-06-11 17:11:16 -0400 (Should run every 20 minutes)
custom_script - started 2012-06-11 17:11:16 -0400 (Should run every 1 hour)
logrotate - started 2012-06-11 17:11:16 -0400 (Job should run every 1 second, but hasn't run since 2012-06-11 17:11:16 -0400)
exceptional - failed 2012-06-11 17:11:16 -0400 (Another freaking nil error)
exceptionally_anonymous - failed 2012-06-11 17:11:16 -0400 (FAILED!!!)
finish_anytime - finished 2012-06-11 17:11:16 -0400 (no particular deadline)
slow - finished 2012-06-11 17:11:16 -0400 (Finished)
quick - finished 2012-06-11 17:11:16 -0400 (Finished, but 4 seconds late!!!)
good task - finished 2012-06-11 17:11:21 -0400 (no particular deadline)
bad task - failed 2012-06-11 17:11:21 -0400 (RuntimeError - AAAAAAAAAAAAAAAAAAAAA)
(Todo, how do we show the colors in github)
To view status from a web-server:
https://localhost:43210
To record a chore in ruby:
# one time config, not needed if server is on localhost
Chore.set_server('hostname',Chore::Constants::DEFAULT_LISTEN_PORT)
Chore.monitor(:task_name) do
# real work
end
This will record the start and finish of the wrapped block.
Even simpler, you can just record the start of a task:
Chore.start(:task_name)
#do stuff
This still requires you to examine timestamps to see if things are running. If you want a better visual clue that things are broken, you can throw in some options. (All .start options can also be passed into .monitor)
# complain if the task hasn't run in more than an hour.
Chore.start(:task_name, :do_every => 3600)
# complain if the task hasn't run in more than an hour, and
# complain more loudly if it hasn't run in two.
Chore.start(:task_name, :do_every => 3600, :grace_period => 3600)
# complain if the task hasn't finished in an hour.
Chore.start(:task_name, :finish_in => 3600)
# make sure to call Chore.finish(:task_name) explicitly later,
# or use Chore.monitor to do so automatically.
# Record error info for known exception
begin
Chore.start(:goofy_task)
raise Errno::ETIMEDOUT # network problems
rescue Errno::ETIMEDOUT => ex
Chore.fail(:goofy_task, :error => "Crappy router probably needs a reboot."
end
# Add status notes, only the last update shows.
Chore.start(:long_task)
# ...
Chore.status(:long_task, "Downloaded the interwebz")
# ...
Chore.status(:long_task, "Ran bayesian classifier")
# ...
# Remove a task from the store on completion, but not if it fails
Chore.monitor("update widget #{widget.id}", :pop => true) do
# ...
end
# Expire an entry in X seconds, whether it's started, finished or
# failed, so that it no longer shows up in output. Useful when
# attaching an id number or pid to the task name.
Chore.monitor("Cron task with pid #{process_id}", :expire_in => 1.day) {}
There is also a command-line tool so you can use the server without having to fire up irb.
johnmudhead:chore grant$ chore-status
test - failed 2012-06-15 17:33:00 -0400 (AAAAAAAAA)
johnmudhead:chore grant$ chore-client --chore test --action pop
johnmudhead:chore grant$ chore-status
johnmudhead:chore grant$
This gem is signed with rubygems-openpgp. You can verify its integrity by running:
gem install chore --trust
Much more information on signing is available at the rubygems-openpgp Certificate Authority.
Signing key:
pub 2048R/E3B5806F 2010-01-11 [expires: 2014-01-03]
Key fingerprint = A530 C31C D762 0D26 E2BA C384 B6F6 FFD0 E3B5 806F
uid Grant T. Olson (Personal email) <[email protected]>
uid Grant T Olson <[email protected]>
uid Grant T. Olson (pikimal) <[email protected]>
sub 2048R/6A8F7CF6 2010-01-11 [expires: 2014-01-03]
sub 2048R/A18A54D6 2010-03-01 [expires: 2014-01-03]
sub 2048R/D53982CE 2010-08-31 [expires: 2014-01-03]