Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow backends to support find-and-lock in a single request #151

Closed
wants to merge 2 commits into from
Closed

Allow backends to support find-and-lock in a single request #151

wants to merge 2 commits into from

Conversation

betamatt
Copy link
Collaborator

This patch adds find-and-lock semantics to the active record backend and modifies the worker to use this method when the backend supports it. The idea is to allow workers to synchronize on the underlying database when the database supports it. As we've scaled out our worker count we've found contention between workers trying to lock jobs is a serious problem and generates a lot of extra query traffic in the current implementation. InnoDB's row level locks can instead be used to synchronize the worker polling so that each worker is guaranteed to get a lockable job, or none at all.

We haven't finished testing this yet but I'd like to start gathering feedback.

@bkeepers
Copy link

I actually just pushed a bunch of changes, one of which is adding a #reserve method. The worker calls this and expects it to find and lock an available job. It still does it in 2 queries though.

@bkeepers
Copy link

If you have time to rebase your changes and submit another pull request, it would be much appreciated.

@betamatt
Copy link
Collaborator Author

Brandon, I'll try to get to it early next week.

@betamatt
Copy link
Collaborator Author

betamatt commented Dec 1, 2010

Rebased: #167

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants