Add jobs doc #140

jinroh · 2016-12-08T16:55:41Z

No description provided.

codecov-io · 2016-12-08T17:00:53Z

Current coverage is 54.38% (diff: 100%)

Merging #140 into master will increase coverage by 2.12%

@@             master       #140   diff @@
==========================================
  Files            28         30     +2   
  Lines          2696       2848   +152   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           1409       1549   +140   
- Misses         1053       1063    +10   
- Partials        234        236     +2

Powered by Codecov. Last update d243b55...e28ab68

nono · 2016-12-08T17:15:31Z

docs/jobs.md

+Cozy Jobs
+=========
+
+Jobs are designed to represent asynchronous tasks that your cozy can execute. These tasks can be scheduled in advance, recurring or suddenn and provide various services.


s/suddenn/sudden/

nono · 2016-12-08T17:16:06Z

docs/jobs.md

+
+At the time, we do not provide "generic" and programmable tasks. This list of available workers is part of the defined API.
+
+The job queue can be either local the cozy-stack when used as a monolitic instance (self-hosted for instance) or distributed via a redis server for distributed infrastructures.


local to the cozy-stack?

nono · 2016-12-08T17:19:58Z

docs/jobs.md

+ - *parallel limitations*: each launcher will have a limited number of workers it is allowed to queue in parallel
+ - *back pressure*: each launcher should have a backpressure policy to drop job spawning when not necessary. For instance: 
+ - in the case of an `@event` launcher, a job doing an external API call should not be spawned if another one is already running for another close event.
+ - when no worker is availabe, or the queue has too many elements, it can decide to drop the job action (given some informations)


s/availabe/available/

nono · 2016-12-08T17:22:54Z

docs/jobs.md

+Jobs can be launched by three types of launchers:
+
+ - `@cron` to schedule recurring jobs, either periodic or scheduled at specific times
+ - `@interval` to schedule jobs executed at a given fix interval


My first impression is that @cron and @interval are 2 ways to do the same thing. What are the differences between them?

nono · 2016-12-08T17:23:33Z

docs/jobs.md

+```
+A duration string is a possibly signed sequence of decimal numbers, each with
+optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". Valid
+time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".


It looks dangerous to accept units below the second.

nono · 2016-12-08T17:33:06Z

docs/jobs.md

+ ]
+}
+```
+


We should probably also have a way to remove a launcher

nono · 2016-12-08T17:35:14Z

docs/jobs.md

+
+Jobs can be launched by three types of launchers:
+
+ - `@cron` to schedule recurring jobs, either periodic or scheduled at specific times


Is it possible to schedule a one-time job? Like send me a notification for this event in my calendar 10 minutes before it start. I don't see how to do that with the @cron syntax below.

nono · 2016-12-08T17:39:03Z

docs/jobs.md

+
+### GET /jobs/queue
+
+List the jobs in the queue.


I think that I'd want to have the number of jobs in queue, before having a list of paginated jobs.

I would like the provide the list associated with metadata about its total size. How would you spec that with JSON-API... ?

You can use meta:

{ "links": { "next": "/jobs/queue?page[cursor]=123123" }, "data": [ { "type": "io.cozy.jobs", "id": "123123", "rev": "1-12334", "attributes": {}, "links": { "self": "/jobs/123123" }, "meta": { "count": 456 } }, ] }

which mean we would have to duplicate the meta informations about the list in each of its elements ?

No, we should probably put in the global meta. Like this:

{ "links": { "next": "/jobs/queue?page[cursor]=123123" }, "data": ["..."], "meta": { "count": 456 } }

nono · 2016-12-08T17:46:59Z

docs/jobs.md

+ - in the case of an `@event` launcher, a job doing an external API call should not be spawned if another one is already running for another close event.
+ - when no worker is availabe, or the queue has too many elements, it can decide to drop the job action (given some informations)
+
+


How do you see the stack taking the jobs? We have n goroutines that can process jobs, and when one of them is free, we take the jobs with the highest priority (or the first arrived in case of egality)? Or we have something more complex with a limit by worker (like no more than 3 thumbnails generation in parallel because it takes a lot of RAM)?

nono · 2016-12-08T17:47:26Z

docs/jobs.md

+ "launcher": "@cron", // "@cron", "@interval", "@event" or ""
+ "launcher_id": "1234", // launcher id, if any
+ "options": {
+ "priority": 3, // priority from 1 to 100


What is the highest priority? 1 or 100?

nono · 2016-12-08T17:48:55Z

Good job! I have some questions & remarks, but it's very good base to construct on.

aenario · 2016-12-08T23:16:58Z

docs/jobs.md

+ - `io.cozy.launchers` for launchers
+
+
+Launchers


I feel triggers is the more used term for this.

aenario · 2016-12-08T23:20:23Z

docs/jobs.md

+Error Handling
+--------------
+
+Jbos can fail to execute their task. We have two ways to parametrize how they should behave in suche cases.


parameterize such

aenario · 2016-12-08T23:22:42Z

docs/jobs.md

+Every launcher has a rate limitation policy to prevent launchers from spawning too many jobs at the same time. The specific rules are not yet decided, but they should work along these two properties:
+
+ - *parallel limitations*: each launcher will have a limited number of workers it is allowed to queue in parallel
+ - *back pressure*: each launcher should have a backpressure policy to drop job spawning when not necessary. For instance: 


This also brings the idea of leading / trailing debounce with @event
ie. a job that do some action after any contact update, but should only be triggered once after several contacts are updated.

Thanks ! Added a comment on throttling / debouncing which is more explicit.

aenario · 2016-12-08T23:26:20Z

docs/jobs.md

+
+### GET /jobs/queue
+
+List the jobs in the queue.


Privacy complication : an app has the permission sendMail, it does not mean it should be able to see all mails in the queue to be sent.

Agreed, we should probably introduce a queue for each application. I'll probably add a "Permissions" paragraph to discuss that.

nono · 2016-12-09T14:53:00Z

I still don't see how to schedule a one-time job in the future:

Is it possible to schedule a one-time job? Like send me a notification for this event in my calendar 10 minutes before it start. I don't see how to do that with the @Cron syntax below.

nono · 2016-12-12T10:08:09Z

docs/jobs.md

+
+ - `@cron` to schedule recurring jobs scheduled at specific times
+ - `@interval` to schedule periodic jobs executed at a given fix interval
+ - `@event` to launch a job after a change in the cozy


This week-end, I thought of a new type of trigger: @webhook. It would create an unguessable URL that we can give to external services for pushing a job. For example, we could give a webhook URL to github to update on our cozy the app we are working on after each git push.

This is nice, I'll add it the spec.

nono · 2016-12-12T15:52:01Z

docs/jobs.md

+Triggers
+--------
+
+Jobs can be launched by three different types of triggers:


s/three/six/

nono · 2016-12-12T15:52:09Z

docs/jobs.md

+ - `@event` to launch a job after a change in the cozy
+ - `@webhook` to launch a job by requesting an unguessable URL
+
+These three triggers have specific syntaxes to describe when jobs should be scheduled. See below for more informations.


s/three/six/

Add jobs doc

d07dc03

jinroh force-pushed the jobs branch from 3aba36c to d07dc03 Compare December 8, 2016 17:04

nono reviewed Dec 8, 2016

View reviewed changes

aenario suggested changes Dec 8, 2016

View reviewed changes

jinroh added 8 commits December 9, 2016 08:19

Typos and part of remarks from PR

e100229

Added comments on throttling / debouncing

b368a66

Add DELETE launcher

0fa2359

Specify priority order

3492213

Add "Worker pool" and "Permission" paragraphs

d0b157e

s/launcher/trigger/

fcc842c

Get rid of worker_id field

98c7468

Fix some typos

1520d31

aenario approved these changes Dec 9, 2016

View reviewed changes

nono reviewed Dec 12, 2016

View reviewed changes

Add @at, @in and @webhook triggers

5752e54

aenario approved these changes Dec 12, 2016

View reviewed changes

nono reviewed Dec 12, 2016

View reviewed changes

Add meta with count information for queue list

e28ab68

nono merged commit bd707ef into cozy:master Dec 13, 2016

jinroh deleted the jobs branch December 13, 2016 08:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add jobs doc #140

Add jobs doc #140

jinroh commented Dec 8, 2016

codecov-io commented Dec 8, 2016 •

edited

Loading

nono Dec 8, 2016

nono Dec 8, 2016

nono Dec 8, 2016

nono Dec 8, 2016

nono Dec 8, 2016

nono Dec 8, 2016

jinroh Dec 9, 2016

nono Dec 8, 2016

nono Dec 8, 2016

jinroh Dec 9, 2016

nono Dec 9, 2016

jinroh Dec 12, 2016

nono Dec 12, 2016

nono Dec 8, 2016

nono Dec 8, 2016

nono commented Dec 8, 2016

aenario Dec 8, 2016

aenario Dec 8, 2016

aenario Dec 8, 2016

jinroh Dec 9, 2016

aenario Dec 8, 2016

jinroh Dec 9, 2016

nono commented Dec 9, 2016

nono Dec 12, 2016

jinroh Dec 12, 2016

nono Dec 12, 2016

nono Dec 12, 2016


		At the time, we do not provide "generic" and programmable tasks. This list of available workers is part of the defined API.

		The job queue can be either local the cozy-stack when used as a monolitic instance (self-hosted for instance) or distributed via a redis server for distributed infrastructures.


		Jobs can be launched by three types of launchers:

		- `@cron` to schedule recurring jobs, either periodic or scheduled at specific times

		- in the case of an `@event` launcher, a job doing an external API call should not be spawned if another one is already running for another close event.
		- when no worker is availabe, or the queue has too many elements, it can decide to drop the job action (given some informations)

Add jobs doc #140

Add jobs doc #140

Conversation

jinroh commented Dec 8, 2016

codecov-io commented Dec 8, 2016 • edited Loading

Current coverage is 54.38% (diff: 100%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nono commented Dec 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nono commented Dec 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Dec 8, 2016 •

edited

Loading