Merge of high priority issue fixes and pull requests on top of version 0.6.2

[behrad]

I did merge in some important fixes and improvements from the community and our own deployment experience of Kue. This merge contains (merging, testing and fixing community code in pull requests):

DONE:

FIX: Suppress "undefined" messages on String errors (#230)
FIX: Disable Search Indexes (redis memory leakage regardless of removing jobs) (related old issue is #58). I tried to merge with #218 but couldn't get it to work.
FIX: Cannot read property id of undefined errors (#252)
FIX: Parameterize number of jobs limit in promotion cycles (#244)
FEATURE: Global graceful shutdown - handy on exiting on critical errors (https://github.com/LOKE/kue)
FEATURE: Pause/resume-ability of specific worker types - called within worker processors (#163)
Enhancement: Singleton Queue
FIX: Ensure event subscription before job save (#179)
FIX: failed event is called in first attempt (#142)

And there are a list of other important issues that can be merged in easily:

TO-BE-DONE:

and these are seemed to be important and potential for next merges (ordered by simplicity)

FEATURE: Passing back results from job processor by job.data (SpiderStrategies@7defce2) (not sure kue really needs this... should be discussed)
Enhancement: Improving QOS, better timing, callback base job functions, ... (inspired by @dfoody extensions)
FIX: Issues when removing failed jobs (#99 and #197)
FEATURE: Delaying failed jobs (#227)

[bulkan]

I'm sorry to say that the likely hood of this PR getting merged is low in light of still open PR's e.g #134

[behrad]

HAPPY NEWS: I got access as a new Kue maintainer @bulkan and will merge some PRs in a few days...

[bulkan]

@behrad thats awesome news ! Been trying to get commit access too.

If you need help with testing & triaging let me know.

[behrad]

@bulkan sure, I wonder if somebody has a test case suite which we can merge! Or write tests.
I will try to merge proven & high priority changes in a few days first.

[nathanbowser]

@behrad I'm thrilled to see activity in kue. I'm using kue in a project that went live two weeks ago, and kue has been the source of 75% of the problems.

Why do you suggest SpiderStrategies@7defce2 might not belong in kue? This is an important feature if you have a parent process scheduling and saving the kue jobs, and other processes processing those jobs. In my case, once a job has completed, the main process needs access to the data the worker fetched.

If you need a hand with anything, please let me know.

[behrad]

@nathanbowser Since you can already do this with built-in APIs using job.data:

How worker saves data:

job.set( 'data' , JSON.stringify(data), done );

How access data in producer:

job.on( 'complete', function() {
   queue.Job.get( job.id, function( err, job ){
       // have fun with job.data
   }
});

However a simpler get method was proposed in job class, a year before as below:

kue.Job.prototype.get = function(key, fn){
    var id = this.id;
    this.client.hget('q:job:' + this.id, key, function( err, res ){
        if( res ) {
            try {
                res = JSON.parse( res );
            } catch( e ) {
                err = e;
            } finally {
                fn && fn( err, res );
            }
        } else {
            fn && fn( err );
        }
    });
    return this;
};

This is how we are passing data between worker and producers. I know this

1. is ugly
2. overrides initial producers data value
3. is not memory scalable

But; the point is, may be a large amount of client code are written without passing results to done callback during the usage of Kue till now overcoming this problem. However it can be added as a neat-nice-new feature!

[nathanbowser]

I totally see what you're saying.

It seems inconsistent to have job.on('failed', function (err) {}) receive the error, but job.on('complete'...) not receive the data. Maybe the failed callback receiving an error is only in my fork, though. I can't remember.

You're also introducing more operations with redis to do the kue.Job.get(...) in your complete callback.

I guess it boils down to personal preference. Should each job have to set some result on the job, and then the producer looks at the field? Or should the worker pass the data back as a result?

[behrad]

Should each job have to set some result on the job, and then the producer looks at the field? Or should the worker pass the data back as a result?

This is what we had been talking about a year ago here, I also wish the second method was available. BTW:
The second looks more loose (no convention between producer/consumer)
The first looks more reliable since Redis lacks durability on pub/sub commands. What if you miss the complete and so the job's result? What if you want to batch process job results or access result later? So the first also have some benefits.
Other than these, the most important for me (and all others using the first method) in real, is the time I should spend to rewrite my existing client code to do the transition to the new API.

otherwise, I love second approach 'cause of its looseness and simplicity of usage.

[behrad]

Anybody ready to do their own tests on a develop branch so I push it?

[victorquinn]

Awesome, thanks for the update @behrad !