-
Notifications
You must be signed in to change notification settings - Fork 137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add jobs doc #140
Add jobs doc #140
Conversation
Current coverage is 54.38% (diff: 100%)@@ master #140 diff @@
==========================================
Files 28 30 +2
Lines 2696 2848 +152
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 1409 1549 +140
- Misses 1053 1063 +10
- Partials 234 236 +2
|
Cozy Jobs | ||
========= | ||
|
||
Jobs are designed to represent asynchronous tasks that your cozy can execute. These tasks can be scheduled in advance, recurring or suddenn and provide various services. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/suddenn/sudden/
|
||
At the time, we do not provide "generic" and programmable tasks. This list of available workers is part of the defined API. | ||
|
||
The job queue can be either local the cozy-stack when used as a monolitic instance (self-hosted for instance) or distributed via a redis server for distributed infrastructures. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
local to the cozy-stack?
- *parallel limitations*: each launcher will have a limited number of workers it is allowed to queue in parallel | ||
- *back pressure*: each launcher should have a backpressure policy to drop job spawning when not necessary. For instance: | ||
- in the case of an `@event` launcher, a job doing an external API call should not be spawned if another one is already running for another close event. | ||
- when no worker is availabe, or the queue has too many elements, it can decide to drop the job action (given some informations) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/availabe/available/
Jobs can be launched by three types of launchers: | ||
|
||
- `@cron` to schedule recurring jobs, either periodic or scheduled at specific times | ||
- `@interval` to schedule jobs executed at a given fix interval |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My first impression is that @cron
and @interval
are 2 ways to do the same thing. What are the differences between them?
``` | ||
A duration string is a possibly signed sequence of decimal numbers, each with | ||
optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". Valid | ||
time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks dangerous to accept units below the second.
] | ||
} | ||
``` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably also have a way to remove a launcher
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
Jobs can be launched by three types of launchers: | ||
|
||
- `@cron` to schedule recurring jobs, either periodic or scheduled at specific times |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to schedule a one-time job? Like send me a notification for this event in my calendar 10 minutes before it start. I don't see how to do that with the @cron
syntax below.
|
||
### GET /jobs/queue | ||
|
||
List the jobs in the queue. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that I'd want to have the number of jobs in queue, before having a list of paginated jobs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like the provide the list associated with metadata about its total size. How would you spec that with JSON-API... ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use meta
:
{
"links": {
"next": "/jobs/queue?page[cursor]=123123"
},
"data": [
{
"type": "io.cozy.jobs",
"id": "123123",
"rev": "1-12334",
"attributes": {},
"links": {
"self": "/jobs/123123"
},
"meta": {
"count": 456
}
},
]
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which mean we would have to duplicate the meta informations about the list in each of its elements ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, we should probably put in the global meta. Like this:
{
"links": {
"next": "/jobs/queue?page[cursor]=123123"
},
"data": ["..."],
"meta": {
"count": 456
}
}
- in the case of an `@event` launcher, a job doing an external API call should not be spawned if another one is already running for another close event. | ||
- when no worker is availabe, or the queue has too many elements, it can decide to drop the job action (given some informations) | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do you see the stack taking the jobs? We have n goroutines that can process jobs, and when one of them is free, we take the jobs with the highest priority (or the first arrived in case of egality)? Or we have something more complex with a limit by worker (like no more than 3 thumbnails generation in parallel because it takes a lot of RAM)?
"launcher": "@cron", // "@cron", "@interval", "@event" or "" | ||
"launcher_id": "1234", // launcher id, if any | ||
"options": { | ||
"priority": 3, // priority from 1 to 100 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the highest priority? 1 or 100?
Good job! I have some questions & remarks, but it's very good base to construct on. |
- `io.cozy.launchers` for launchers | ||
|
||
|
||
Launchers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel triggers is the more used term for this.
Error Handling | ||
-------------- | ||
|
||
Jbos can fail to execute their task. We have two ways to parametrize how they should behave in suche cases. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
parameterize such
Every launcher has a rate limitation policy to prevent launchers from spawning too many jobs at the same time. The specific rules are not yet decided, but they should work along these two properties: | ||
|
||
- *parallel limitations*: each launcher will have a limited number of workers it is allowed to queue in parallel | ||
- *back pressure*: each launcher should have a backpressure policy to drop job spawning when not necessary. For instance: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This also brings the idea of leading / trailing debounce with @event
ie. a job that do some action after any contact update, but should only be triggered once after several contacts are updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks ! Added a comment on throttling / debouncing which is more explicit.
|
||
### GET /jobs/queue | ||
|
||
List the jobs in the queue. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Privacy complication : an app has the permission sendMail, it does not mean it should be able to see all mails in the queue to be sent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, we should probably introduce a queue for each application. I'll probably add a "Permissions" paragraph to discuss that.
I still don't see how to schedule a one-time job in the future:
|
|
||
- `@cron` to schedule recurring jobs scheduled at specific times | ||
- `@interval` to schedule periodic jobs executed at a given fix interval | ||
- `@event` to launch a job after a change in the cozy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This week-end, I thought of a new type of trigger: @webhook
. It would create an unguessable URL that we can give to external services for pushing a job. For example, we could give a webhook URL to github to update on our cozy the app we are working on after each git push.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is nice, I'll add it the spec.
Triggers | ||
-------- | ||
|
||
Jobs can be launched by three different types of triggers: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/three/six/
- `@event` to launch a job after a change in the cozy | ||
- `@webhook` to launch a job by requesting an unguessable URL | ||
|
||
These three triggers have specific syntaxes to describe when jobs should be scheduled. See below for more informations. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/three/six/
No description provided.