Iron Cushion is a benchmark and load testing tool for CouchDB, developed by Ad Hoc Labs, Inc. It proceeds in two steps: First, documents are bulk inserted using CouchDB's Bulk Document API. Second, documents are individually created, read, updated, and deleted with random ordering of operations using CouchDB's Document API. Below we refer to the former as the "bulk insert step," and the latter as the "CRUD operations step." Statistics for both steps are recorded separately and displayed afterward.
It is written in Java for version 5.0 and higher, depends only on the Netty library, and is released under the MIT license.
database_address
: The address of the database, of the formhttps://[hostname]:[port]
orhttps://[ipaddress]:[port]
.database_name
: The name of the database.num_connections
: The number of concurrent connections to establish to the database.json_document_schema_filename
: A file containing JSON describing the schema of documents created during the benchmark.xml_document_schema_filename
: A file containing XML describing the schema of documents created during the benchmark.seed
: An optional integer flag that specifies the seed to use for the random number generator.
Either json_document_schema_filename
or xml_document_schema_filename
must be provided. For details on the contents of these files, see "Document Generation" below.
The following flags control the bulk insert step:
num_documents_per_bulk_insert
: The number of documents in each bulk insert operation.num_bulk_insert_operations
: The number of bulk insert operations performed by each connection.
For example, if num_connections
is 50
, num_documents_per_bulk_insert
is 1000
, and num_bulk_insert_operations
is 20
, then after the bulk insert step there will be 50 x 1,000 x 20 = 1,000,000 documents in the database.
The following flags control the CRUD operations step:
num_crud_operations
: The number of CRUD operations performed by each connection.create_weight
: Weight defining the number of create operations relative to other operations.read_weight
: Weight defining the number of read operations relative to other operations.update_weight
: Weight defining the number of update operations relative to other operations.delete_weight
: Weight defining the number of delete operations relative to other operations.
For example, if create_weight
is 2
, read_weight
is 3
, update_weight
is 2
, and delete_weight
is 1
, then 2/8 of all CRUD operations will be create operations, 3/8 of all CRUD operations will be read operations, 2/8 of all CRUD operations will be update operations, and 1/8 of all CRUD operations will be delete operations. If num_crud_operations
is 10000
, this equals 2,500 create operations, 3,750 read operations, 2,500 update operations, and 1,250 delete operations per connection.
Every update or delete operation requires the _rev
value of a document. Such a value comes from either reading the document from the database earlier, or from creating the document earlier and recording the returned value. Therefore the sum create_weight + read_weight
must be greater than or equal to delete_weight
. Additionally, if update_weight
is greater than 0
, then create_weight + read_weight
must be greater than 0
. If these inequalities don't hold, the flags fail validation. Finally, if delete_weight
is large enough such that the number of documents to be deleted exceeds the sum of number of documents bulk inserted and the number of documents created from CRUD operations, the flags fail validation. To remedy this, bulk insert more documents, increase create_weight
, or decrease delete_weight
.
Note that while CouchDB is schemaless, Iron Cushion requires a schema to serve as a template for generated documents that are inserted during the bulk insert step, or inserted or updated during the CRUD operations step. This allows the user to easily control their level of complexity. A schema can be defined either using JSON or XML, but you will likely find the former easier.
Subject to the quality of the pseudo-random number generator, generated values adhere to the following rules:
- Boolean values have the same likelihood of being
true
orfalse
. - Integer values follow a uniform distribution.
- Floating point values follow a uniform distribution between
0.0
and1.0
. - Strings are between 1 and 5 words in length, where adjacent words are separated by a space. Words are chosen from a dictionary of 32,768 words. Each word is between 3 and 15 characters in length and follows the alphabet
[A-Za-z0-9?!]
. The chosen number of words in a string, the length of each word, and each character in each word all follow uniform distributions.
The json_document_schema_filename
command line flag specifies a file containing JSON that defines a schema for documents in the database.
A new document is generated from the schema by the following rules:
- Any boolean value read is replaced by a randomly generated boolean value.
- Any integer value read is replaced by a randomly generated integer value.
- Any floating point value read is replaced by a randomly generated floating point value.
- Any string value read is replaced by a randomly generated string.
All other properties of the JSON, such as array lengths, value names in objects, null
values, and nested types are preserved. The advantage of using a JSON schema is that any document stored in CouchDB is a valid schema for Iron Cushion. Iron Cushion will automatically remove the special _id
and _rev
values from the JSON schema provided.
An example can be found in iron-cushion/iron-cushion/data/example_schema.json
. Its schema is equivalent to the one in iron-cushion/iron-cushion/data/example_schema.xml
.
The xml_document_schema_filename
command line flag specifies an file containing XML that defines a schema for documents in the database. In the future, the XML file may allow adding attributes to these tags to specify properties like minimum and maximum values for generated integers, etc.
There are seven principal tags, and the outer-most tag must be <object>
:
- The
<object>
tag translates to a JSON object containing name-value pairs. The<object>
tag encloses 0 or more<entry>
tags to define name-value pairs. Each<entry>
tag contains a<name>
and<value>
tag. The name of the value, which must be a string, is enclosed by the<name>
tag. The type of the value is enclosed by the<value>
tag. - The
<array>
tag translates to an JSON array. The<array>
tag encloses 0 or more<element>
tags to define its elements. The type of each element is enclosed by its<element>
tag. - The
<boolean />
tag translates to a generated boolean value. - The
<integer />
tag translates to an generated integer value. - The
<float />
tag translates to a generated floating point value. - The
<string />
tag translates to a generated string. - The
<null />
tag translates to anull
value.
An example can be found in iron-cushion/iron-cushion/data/example_schema.xml
. Its schema is equivalent to the one in iron-cushion/iron-cushion/data/example_schema.json
.
To update to a document during the CRUD operations step, Iron Cushion regenerates and replaces a randomly chosen value from the document's top level object. The updated document is sent to CouchDB using a PUT
request.
The following schema, found in file iron-cushion/iron-cushion/data/example_schema.json
:
{
"array1": [
[
0.0,
0.0
],
{},
true,
null
],
"boolean1": true,
"integer1": 0,
"obj1": {
"array2": [],
"obj2": {
"boolean2": true
}
},
"string1": ""
}
Can generate the following document:
{
"array1": [
[
0.8474894,
0.30425853
],
{},
false,
null
],
"boolean1": true,
"integer1": 1929847379,
"obj1": {
"array2": [],
"obj2": {
"boolean2": false
}
},
"string1": "928lR8eM7DcBSgR 598A8VxzeFE2 uKTF FqiMEmxdLJmDni"
}
If you don't want to go through the hassle of running javac
yourself, simply copy the files iron-cushion/iron-cushion/dist/IronCushion-0.1.jar
and iron-cushion/iron-cushion/lib/netty-3.3.1.Final.jar
to a directory outside of the iron-cushion
project. Pull up the command line, cd
into that directory, and run the following:
java -cp IronCushion-0.1.jar:netty-3.3.1.Final.jar co.adhoclabs.ironcushion.Benchmark [flags]
Where [flags]
is replaced with the Iron Cushion command line flags of your choosing.
The following flags specify using 100 connections, collectively bulk inserting 2,000,000 documents, followed by performing 20,000 create operations, 20,000 read operations, 30,000 update operations, and 30,000 delete operations.
--num_connections=100
--num_documents_per_bulk_insert=1000
--num_bulk_insert_operations=20
--num_crud_operations=1000
--create_weight=2
--read_weight=2
--update_weight=3
--delete_weight=3
Running the benchmark program with these flags on my 1.83 GHz Intel Core Duo MacBook, CouchDB on my Intel Core 2 2.83GHz quad-core desktop, and across my 100Mbit home LAN, I get the results below.
BULK INSERT BENCHMARK RESULTS:
timeTaken=249.182 secs
totalJsonBytesSent=374,240,177 bytes
totalJsonBytesReceived=138,823,936 bytes
localProcessing={min=1.363 secs, max=2.906 secs, median=1.800 secs, sd=0.323 secs}
sendData={min=9.066 secs, max=29.611 secs, median=19.002 secs, sd=4.287 secs}
remoteProcessing={min=171.507 secs, max=214.598 secs, median=203.845 secs, sd=10.918 secs}
receiveData={min=4.933 secs, max=23.565 secs, median=11.875 secs, sd=3.856 secs}
remoteProcessingRate=10,003.030 docs/sec
localInsertRate=8,676.693 docs/sec
timeTaken
is how long it took for the slowest connection to complete all bulk inserts.totalJsonBytesSent
is the number of bytes of JSON sent to CouchDB, and therefore does not include bytes from HTTP headers.totalJsonBytesReceived
is the number of bytes of JSON received from CouchDB, and therefore does not include bytes from HTTP headers.localProcessing
is how much time each connection spent preparing and encoding sent JSON and decoding received JSON.sendData
is how much time each connection spent sending data to CouchDB.remoteProcessing
is how much time each connection spent waiting for the beginning of responses from CouchDB after sending bulk insert messages.receiveData
is how much time each connection spent receiving data from CouchDB.remoteProcessingRate
is the rate at which CouchDB adds documents in bulk, i.e. the total number of documents bulk inserted divided by the sum of allremoteProcessing
times.localInsertRate
is the rate at which the benchmark added documents in bulk, i.e. the total number of documents bulk inserted divided by the sum of allsendData
,remoteProcessing
, andreceiveData
times.
CRUD BENCHMARK RESULTS:
timeTaken=84.654 secs
totalJsonBytesSent=10,646,425 bytes
totalJsonBytesReceived=10,704,882 bytes
localProcessing={min=0.002 secs, max=0.062 secs, median=0.016 secs, sd=0.010 secs}
sendData={min=0.000 secs, max=0.035 secs, median=0.002 secs, sd=0.005 secs}
remoteCreateProcessing={min=20.070 secs, max=22.376 secs, median=21.113 secs, sd=0.464 secs}
remoteReadProcessing={min=1.816 secs, max=2.566 secs, median=2.236 secs, sd=0.135 secs}
remoteUpdateProcessing={min=29.215 secs, max=31.504 secs, median=30.425 secs, sd=0.520 secs}
remoteDeleteProcessing={min=29.357 secs, max=31.609 secs, median=30.602 secs, sd=0.503 secs}
remoteCreateProcessingRate=949.141 docs/sec
remoteReadProcessingRate=9,015.862 docs/sec
remoteUpdateProcessingRate=980.172 docs/sec
remoteDeleteProcessingRate=980.154 docs/sec
timeTaken
is how long it took for the slowest connection to complete all CRUD operations.totalJsonBytesSent
is the number of bytes of JSON sent to CouchDB, and therefore does not include bytes from HTTP headers.totalJsonBytesReceived
is the number of bytes of JSON received from CouchDB, and therefore does not include bytes from HTTP headers.localProcessing
is how much time each connection spent preparing and encoding sent JSON and decoding received JSON.sendData
is how much time each connection spent sending data to CouchDB.remoteCreateProcessing
is how much time each connection spent waiting for a response from CouchDB after sending messages for create operations.remoteReadProcessing
is how much time each connection spent waiting for a response from CouchDB after sending messages for read operations.remoteUpdateProcessing
is how much time each connection spent waiting for a response from CouchDB after sending messages for update operations.remoteDeleteProcessing
is how much time each connection spent waiting for a response from CouchDB after sending messages for delete operations.remoteCreateProcessingRate
is the rate at which CouchDB creates documents, i.e. the total number of documents created divided by the sum of allremoteCreateProcessing
times.remoteReadProcessingRate
is the rate at which CouchDB reads documents, i.e. the total number of documents read divided by the sum of allremoteReadProcessing
times.remoteUpdateProcessingRate
is the rate at which CouchDB updates documents, i.e. the total number of documents updated divided by the sum of allremoteUpdateProcessing
times.remoteDeleteProcessingRate
is the rate at which CouchDB deletes documents, i.e. the total number of documents deleted divided by the sum of allremoteDeleteProcessing
times.