Skip to content
/ wrek Public

A little Erlang library for executing task dependency graphs

License

Notifications You must be signed in to change notification settings

rkallos/wrek

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

59 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

wrek

Author: Richard Kallos

Lightweight concurrent DAG execution engine

Description

Wrek is designed to execute task dependency graphs concurrently. A vertex V in a task dependency graph can execute once all vertices with paths to V have finished executing. This execution model is similar to tools like Concourse CI.

To retrieve events from wrek, you can pass in the pid of a gen_event process, and add handlers as you see fit.

Requirements

  • Erlang 19.0+

Build

$ rebar3 compile

How to use

  1. Write callback modules that implement the wrek_vert behaviour.
  2. Create a map Map reflecting the structure of the graph you want to run.
  3. wrek:start(Map) or wrek:start(Map, Opts).

Options

  • {event_manager, pid()}: Specify a gen_event process to forward events to
  • {failure_mode, partial | total} (default: total): Switch between partial and total failure modes. Total failure will immediately shut down all running vertices within a DAG. Partial failure will cancel running all tasks reachable by any failed vertex, but will continue until all vertices finish running.
  • {global_timeout, integer() | undefined}: Specify a timeout in milliseconds by which the entire DAG will shutdown if it hasn't already finished. An undefined value means no timeout. (default: undefined)

Example

-module(true_vert).
-behaviour(wrek_vert).

-export([run/2]).

run(_Args, Parent) ->
    {ok, Fun} = wrek_vert:exec(Parent, ".", "true"),
    ok = Fun(),
    {ok, #{}}.
1> Map = #{
    one => #{module => true_vert, args => [], deps => []},
    two => #{module => true_vert, args => [], deps => [one]},
    three => #{module => true_vert, args => [], deps => [one]}}.
2> wrek:start(Map). % Runs one, then two+three concurrently

Individual Vertex Timeout

VertDefn = #{module => wrek_sleep_vert, args => [], deps => [], timeout => 10},

Where timeout is the number of milliseconds until normal completion or forced timeout.