An automated testing tool for comparing visual differences between two versions of a website.
Kontrast lets you build a test suite to run against your test and production websites. It uses Selenium to take screenshots and ImageMagick to compare them. Kontrast then produces a detailed gallery of its test results.
-
Ruby 2.0+
-
Install ImageMagick. You can do this on OS X via brew with:
$ brew install imagemagick
-
Make sure you have Firefox or a different Selenium-compatible browser installed. By default, Firefox is used.
Add this line to your application's Gemfile:
gem 'kontrast'
And then execute:
$ bundle
Or install it yourself as:
$ gem install kontrast
Lastly, generate the config file:
$ kontrast generate_config
If you're in Rails, the config file will be generated in config/initializers/kontrast.rb
.
Otherwise, the config file will be generated in your current directory.
Here's all the config you need to get started:
Kontrast.configure do |config|
# Set your test and production domains
config.test_domain = "http:https://localhost:3000"
config.production_domain = "http:https://www.example.com"
# Build your test suite
# These pages will open in a 1280px-wide browser
config.pages(1280) do |page|
page.home "/"
page.about "/about"
end
# These pages will open in a 320px-wide browser
config.pages(320) do |page|
page.home "/"
page.about "/about"
end
end
Run Kontrast (use bundle exec
and omit the --config flag if you're within a Rails app):
$ kontrast local_run --config ./kontrast_config.rb
...
...
...
Kontrast is all done!
You can find the gallery at: /tmp/shots/1410309651/gallery/gallery.html
Review the gallery in your Favorite Browser:
$ open /tmp/shots/1410309651/gallery/gallery.html
We designed Kontrast from the very beginning to work with multiple nodes. At Harry's, we use CircleCI for testing and Kontrast works perfectly with CircleCI's multi-container features.
Because we ultimately need to generate a gallery with all test results from all given nodes, Kontrast uploads the test images it creates plus a per-node manifest file to S3. After all the tests have run, a single node downloads the manifest files and parses them to create a single gallery.
Here's how to get set up:
config.run_parallel = true
Set how many nodes you have in total and the zero-based index of the current node. Kontrast will automatically split up tests among these nodes.
config.total_nodes = 6
config.current_node = 2
Set your S3 details:
config.aws_bucket = "kontrast-test-results"
config.aws_key = ENV['AWS_KEY']
config.aws_secret = ENV['AWS_SECRET']
Set the local path where output images will be stored before they are uploaded to S3. This is also where the gallery will be saved on the node that runs the make_gallery
command. This path will be created if it doesn't already exist.
config.local_path = "tmp/kontrast"
Set the remote path relative to your S3 bucket's root where Kontrast's output files will be uploaded to. It should be unique to every test.
config.remote_path = "artifacts.#{ENV['BUILD_NUMBER']}"
This command should run in parallel on every node. Use bundle exec
and omit the --config flag if your app is bundle
'd along with Rails.
$ kontrast run_tests --config /path/to/config.rb
This command should only run on one node after all the other nodes have completed the previous command. Use bundle exec
and omit the --config flag if your app is bundle
'd along with Rails.
$ kontrast make_gallery --config /path/to/config.rb
At this point, the gallery should be saved to config.local_path
and uploaded to config.remote_path
. Check it out in your Favorite Browser.
Here's an example of how to run Kontrast within a Rails app using CircleCI:
test:
post:
- bundle exec rails server:
background: true
parallel: true
- bundle exec kontrast run_tests:
parallel: true
- bundle exec kontrast make_gallery
If you want Kontrast to exit with an error code (and fail your build) if an exception is raised while running, use this option:
config.fail_build = true
Choose which Selenium driver you'd like to use. Kontrast has only been tested on the default Firefox driver but we would love feedback and/or pull requests for other drivers.
config.browser_driver = "firefox"
You may set a driver's profile options in this hash.
config.browser_profile = {
"general.useragent.override" => "Some Cool Kontrast User Agent",
"image.animation_mode" => "none"
}
See http:https://www.imagemagick.org/RMagick/doc/constants.html#MetricType for available values.
config.distortion_metric = "MeanAbsoluteErrorMetric"
The ImageMagick comparison tool emphasizes differences with this color. Valid options are an RMagick color name or pixel.
config.highlight_color = "blue"
The ImageMagick comparison tool deemphasizes differences with this color. Valid options are an RMagick color name or pixel.
config.lowlight_color = "rgba(255, 255, 255, 0.3)"
To make Kontrast even more powerful, we provided a set of hooks that you can use in your configuration.
Runs before the entire suite.
config.before_run do
WebMock.disable!
end
Runs after the entire suite.
config.after_run do
WebMock.enable!
end
Runs before the gallery creation step.
config.before_gallery do
WebMock.disable!
end
Runs after the gallery creation step.
config.after_gallery do |diffs, gallery_path|
# diffs is a hash containing all the differences that Kontrast found in your test suite
# gallery_path is where Kontrast saved the gallery
end
Runs on every test before Selenium takes a screenshot.
config.before_screenshot do |test_driver, production_driver, test_info|
# test_driver and production_driver are instances of Selenium::WebDriver that you can control
# test_info is a hash with the current test's name and width
end
Runs on every test after Selenium takes a screenshot.
config.after_screenshot do |test_driver, production_driver, test_info|
# same variables are available as with before_screenshot
end
Kontrast's hooks allow you to insert custom functionality into many parts of the test suite. Here are some examples of how we use hooks at Harry's:
Once a build finishes, we let HipChat know if Kontrast found any diffs using the hipchat
gem:
config.after_gallery do |diffs, gallery_path|
hipchat_room = "Kontrast Results"
hipchat_user = "KontrastBot"
if !diffs.empty?
msg = "Kontrast Diffs: #{diffs.keys.join(', ')}. Don't push to production without reviewing these. You can find the gallery at #{gallery_path}."
client = HipChat::Client.new(ENV["HIPCHAT_TOKEN"])
client[hipchat_room].send(hipchat_user, msg, :color => "red")
end
end
Testing our cart page required a bit more setup before we could take a screenshot of it:
config.before_screenshot do |test_driver, production_driver, test|
if test[:name] == "cart"
# prepare our cookie value
cookie_value = super_secret_magic_cart_cookie
# write cookies using Mootools
# http:https://mootools.net/docs/core/Utilities/Cookie
test_driver.execute_script("Cookie.write('cart', '#{cookie_value}');")
production_driver.execute_script("Cookie.write('cart', '#{cookie_value}');")
# refresh the page
test_driver.navigate.refresh
production_driver.navigate.refresh
end
end
You may want to append a URL param to the end of every test path. To avoid doing something like this:
config.pages(1280) do |page|
page.home "/?mobile=1"
page.about "/about?mobile=1"
end
you can do this instead:
config.pages(1280, { mobile: 1 }) do |page|
page.home "/"
page.about "/about"
end
To avoid cluttering up the Kontrast config with lots of per-test hook logic, we made an easy way for you to specify per-test hooks. We do this with specs, which are RSpec-inspired files that contain hooks which only run with their respective tests.
The name of a spec is passed into the Kontrast#describe
method. This spec is automatically bound to any test whose name includes the name of the spec. In the example below, this spec would only run on the 1280_home
test. But if you name your spec home
, it will run on both the 320_home
and 1280_home
tests.
# 1280_home_spec.rb
Kontrast.describe("1280_home") do |spec|
spec.before_screenshot do |test_driver, production_driver, test|
# Do some stuff before screenshotting the 1280_home page
end
spec.after_screenshot do |test_driver, production_driver, test|
# Do some stuff after screenshotting the 1280_home page
end
end
Spec files should go into the ./kontrast_specs
folder by default and must end with _spec.rb
. You can tell Kontrast to look for specs in a different path using the --specs-path
flag. If you're using Rails and don't include the --specs-path
flag, the specs should go in "#{Rails.root}/kontrast_specs"
.
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request