added intentional failures to some doxbee-sequential tests

[alubbe]

As I was discussing bluebird's extreme performance with other developers, they expressed doubt at whether bluebird could be fast when it has do error-catching as opposed to just passing successful promises around.

So I put together a preliminary modification of the three most important tests to give you a possible suggestion on how to make the test more powerful. Here are the old and the new results, using a 50% failure rate:

Old

results for 10000 parallel executions, 1 ms per I/O op

file time(ms) memory(MB)
callbacks-baseline.js 425 34.42
promises-bluebird-generator.js 577 38.68
promises-bluebird.js 682 38.86

Platform info:
Linux 3.11.0-13-generic x64
Node.JS 0.11.8
V8 3.21.18.3
Intel(R) Core(TM) i5-3337U CPU @ 1.80GHz × 4

New

results for 10000 parallel executions, 1 ms per I/O op

file time(ms) memory(MB)
promises-bluebird.js 946 41.25
promises-bluebird-generator.js 1017 34.16
callbacks-baseline.js 1411 38.69

Platform info:
Linux 3.11.0-13-generic x64
Node.JS 0.11.8
V8 3.21.18.3
Intel(R) Core(TM) i5-3337U CPU @ 1.80GHz × 4

Let me know what you think!

[alubbe]

If you think this kind of additional performance testing makes sense, I would go ahead and modify the other tests as well. It's really quite interesting to play with the failure rate and look at the performance hit. I recommend playing with 0%, 5% and 10%

[petkaantonov]

Looks good to me :) However these should be placed in their new folder like doxbee-sequential-errors or such.

[alubbe]

Done.

The results are interesting - all promise libraries take only ~200ms longer than before. But for some, especially kew, memory goes up realllly high.

One thing I find very weird however, is that running the new tests with an error likelihood of 0% yields results that are a lot worse than the tests without any failures. My guess is that this because of the 70k something random numbers generated but maybe you have an idea about that as well?

Here are some numbers that I am getting:

Original:

results for 10000 parallel executions, 1 ms per I/O op

file time(ms) memory(MB)
callbacks-baseline.js 433 34.53
promises-bluebird-generator.js 570 38.43
promises-bluebird.js 664 39.67
callbacks-caolan-async-waterfall.js 1113 71.98
promises-obvious-kew.js 2305 138.18
promises-calvinmetcalf-liar.js 4789 261.28
promises-medikoo-deferred.js 5541 338.42
promises-cujojs-when.js 8085 285.63
promises-tildeio-rsvp.js 11610 307.85
promises-kriskowal-q.js 33417 711.92

Platform info:
Linux 3.11.0-13-generic x64
Node.JS 0.11.8
V8 3.21.18.3
Intel(R) Core(TM) i5-3337U CPU @ 1.80GHz × 4

New:

results for 10000 parallel executions, 1 ms per I/O op

file time(ms) memory(MB)
promises-bluebird-generator.js 703 39.31
promises-bluebird.js 824 40.28
callbacks-baseline.js 1026 38.47
callbacks-caolan-async-waterfall.js 1132 71.43
promises-obvious-kew.js 2310 203.38
promises-calvinmetcalf-liar.js 4838 378.39
promises-cujojs-when.js 8293 362.11
promises-tildeio-rsvp.js 11033 367.17
promises-kriskowal-q.js 36198 889.03
promises-medikoo-deferred.js OOM OOM

results for 10000 parallel executions, 1 ms per I/O op
Likelihood of rejection: 0.1

file time(ms) memory(MB)
promises-bluebird-generator.js 663 38.11
promises-bluebird.js 804 40.39
callbacks-caolan-async-waterfall.js 1138 71.53
callbacks-baseline.js 1154 38.03
promises-obvious-kew.js 2351 203.33
promises-calvinmetcalf-liar.js 4859 281.15
promises-cujojs-when.js 9523 482.37
promises-tildeio-rsvp.js 11135 505.48
promises-kriskowal-q.js 34431 889.31
promises-medikoo-deferred.js OOM OOM

results for 10000 parallel executions, 1 ms per I/O op
Likelihood of rejection: 0.5

file time(ms) memory(MB)
promises-bluebird-generator.js 656 38.42
promises-bluebird.js 809 40.27
callbacks-baseline.js 1041 38.38
callbacks-caolan-async-waterfall.js 1143 71.56
promises-obvious-kew.js 2456 156.38
promises-calvinmetcalf-liar.js 4740 280.31
promises-cujojs-when.js 8123 479.27
promises-tildeio-rsvp.js 10421 357.03
promises-kriskowal-q.js 34799 882.07
promises-medikoo-deferred.js OOM OOM

[petkaantonov]

In promises examples you seem not have defined triggerIntentionalError so they all fail very early from the ReferenceError.

Also in callbacks baseline you are not checking if error was triggered and bailing out but continuing to run all the code even though it is unnecessary.

[alubbe]

that shouldn't be a problem, it's a globally accessible variable

global.triggerIntentionalError = function(){
  if(LIKELIHOOD_OF_REJECTION && Math.random() <= LIKELIHOOD_OF_REJECTION) throw new Error("intentional failure");
 }

[petkaantonov]

Ah, didn't notice it I only saw the local var statement in callbacks. Anyway I think the error thrown is making promise examples run much faster than callbacks-baseline because they will jump directly to the error handlers, skipping all the then steps.. Whereas I think the callbacks example should stop if the intentional error was triggered. We don't want to see benchmarks where the baseline is slower than others :D

[alubbe]

baseline is now fixed. Have a play with the test and let me know if you are finding any interesting new results :)

[petkaantonov]

Thanks, seems to work good now :)