Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recurring Tasks #4

Merged
merged 45 commits into from
Feb 28, 2020
Merged

Recurring Tasks #4

merged 45 commits into from
Feb 28, 2020

Conversation

lloydmeta
Copy link
Owner

@lloydmeta lloydmeta commented Feb 27, 2020

Adds initial support for recurring Tasks.

Recurring Tasks are Tasks that can be defined to automatically enqueue according to a given schedule expression.

Currently, the standard Cron parser from github.com/robfig/cron is used and so it supports basic macros.

The current implementation involves having a given leader server poll for changes, and (re-using the existing leader lock mechanism) depends on mark-as-dirty and soft-deletes of RecurringTasks. The periodic poll for changed-but-unseen data and a secondary polling sync-enforcer (check against all the recurring tasks defined in the data store) ensures that changes to recurring Tasks (creates, deletes, or updates), are taken into account. The polling intervals are configurable.

Each Task enqueued from a Recurring Task also has a recurring_task_id for tracking.

At the moment, there is no support for "do not enqueue if there is a previous un-handled Task" for Recurring Tasks.

@codecov-io
Copy link

codecov-io commented Feb 27, 2020

Codecov Report

Merging #4 into master will increase coverage by 4.67%.
The diff coverage is 80.03%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master       #4      +/-   ##
==========================================
+ Coverage   72.68%   77.36%   +4.67%     
==========================================
  Files          12       18       +6     
  Lines         681     1109     +428     
==========================================
+ Hits          495      858     +363     
- Misses        144      192      +48     
- Partials       42       59      +17
Impacted Files Coverage Δ
internal/domain/task/service.go 0% <ø> (ø) ⬆️
internal/domain/task/recurring/service.go 0% <0%> (ø)
internal/api/controllers/task/controller.go 97.45% <100%> (ø)
...a/server/routing/tasks/recurring/routes_handler.go 100% <100%> (ø)
internal/domain/task/recurring/models.go 100% <100%> (ø)
internal/domain/task/recurring/mock_service.go 35.71% <35.71%> (ø)
internal/api/models/task/models.go 94.2% <66.66%> (+12.18%) ⬆️
internal/domain/task/recurring/manager.go 76.84% <76.84%> (ø)
internal/infra/server/binding/validation/fields.go 84.61% <80%> (ø) ⬆️
...ernal/infra/server/routing/tasks/routes_handler.go 83.94% <85.18%> (ø)
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0d8687...5b2eeae. Read the comment docs.

We'll need to this to decide how we want to run things inside the recurring jobs scheduling loop.

Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
* Move some shared models around from tasks.Service

Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
* Add parsing of ScheduleExpressions with tests

Signed-off-by: lloydmeta <[email protected]>
…l, so that

  we can see newly-deleted recurring tasks and unschedule them.
  * update implementation and add tests
* Begin implementing recurring.Manager

Signed-off-by: lloydmeta <[email protected]>
* Add logging statements to scheduler

Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
* Do not send loaded if the array is zero sized
* Bulk load error handling

Signed-off-by: lloydmeta <[email protected]>
Signed-off-by: lloydmeta <[email protected]>
  * Add thing wrappers around tracers and transactions
  * Use them for background tasks

Signed-off-by: lloydmeta <[email protected]>
* Make service.All reflect "realtime" data by refreshing the index first
* In `enforceSync`, try an optimistic sync of just the problematic things before
  doing a full sync
* More logs tightening

Signed-off-by: lloydmeta <[email protected]>
* Make a shared mock recurring.Service; make the existing test use it
* Add a test for recurring.NewManager

Signed-off-by: lloydmeta <[email protected]>
@lloydmeta lloydmeta marked this pull request as ready for review February 28, 2020 07:34
…Recurring Task

  caused it to be spawned.
  * This required moving the recurring.Id model to task.RecurringTaskId because Go doesn't
    allow "circular" imports
  * Note that the user-facing API model does not have this field at all when creating Tasks.
  * Update ES template and Kibana index pattern to have this field

Signed-off-by: lloydmeta <[email protected]>
@lloydmeta lloydmeta merged commit df2a385 into master Feb 28, 2020
@lloydmeta lloydmeta deleted the feature/recurring-tasks branch February 28, 2020 08:42
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants