(WIP) RFC: Tasks v1 (Multi-step workflows for AI agents)

[creatorrr]

Sharing the tasks specification, semantic and examples here. Comments and suggestions welcome.

Idea

A lot of problems can be broken down into a series of steps with varying degree of involvement of an AI model.

Proposed abstraction:

• Agents:
• Tasks: State machine specification
• Executions: An actual instance of the state machine
• Transitions: Steps that the execution goes through and their entire history

Simplified example

name: some_name
description: Optional Description

# list of tool names referenced
tools_available:
- <tool-uuid>

# or, tools_available: all

input_schema:
# ... json schema of parameters

workflow1:   # Can be any alphanumeric+underscore name
# ... workflow def (list of steps as shown in main below)
workflow2:   # Can be any alphanumeric+underscore name
# ... workflow def

main:   # The workflow that's called when the task is executed
- prompt: 
    - role: system
       content: # jinja template
       ...
  settings: chat completion settings
- evaluate: object with keys and CEL
- workflow: workflow1  # any workflow except main
  arguments:
    arg1: object with keys and CEL
- tool: tool-uuid
  arguments:
    arg1: object with keys and CEL
- error: <message>  # If this is set then transition to error state with message
- if: ...  # CEL
  then:
    workflow: workflow1  # workflow name, any except main
    arguments:
      some_key: CEL
  else:
    workflow: workflow2  # workflow name
    
## planned ##
# - sleep: ...
# - wait_for_input: ...
# - map / reduce: ...
# - search: ...

# prompt jinja template has the following variables:
# - agent, user, session
# (user is optional; in the future session may be optional)
# - inputs[0], ... # 0 being the input of the main workflow
# - tools  # list of tools with name, metadata etc

Pseudocode

class Task:
  id: Uuid
  created_at: datetime
  updated_at: datetime
  spec: TaskSpec
  agent_id: Uuid

class InputSchema(JsonSchema): pass

CanonicalName = StringRegex(r"^[a-z]\w+$")
CEL = ... # Valid CEL Expression

class CELArguments:
  keys: list[CanonicalName]
  values: list[CEL]

class PromptWorkflowStep:
  prompt: list[ChatMLMessage]
  settings: ChatSettings

class EvaluateWorkflowStep:
  evaluate: CELArguments

class YieldWorkflowStep:
  workflow: CanonicalName
  arguments: CELArguments

class ToolCallWorkflowStep:
  tool_id: Uuid
  arguments: CELArguments

class ErrorCallWorkflowStep:
  error: str

class IfElseWorkflowStep:
  if_: CEL
  then: YieldWorkflowStep
  else_: YieldWorkflowStep

WorkflowStep = (
  PromptWorkflowStep
  | EvaluateWorkflowStep
  | YieldWorkflowStep
  | ToolCallWorkflowStep
  | ErrorWorkflowStep
  | IfElseWorkflowStep
)

class TaskWorkflow:
  name: CanonicalName
  steps: list[WorkflowStep]

class TaskSpec:
  name: CanonicalName
  description: str
  tools_available: list[Uuid] | Literal['all']
  input_schema: InputSchema
  workflows: list[TaskWorkflow]

# Validations
# - workflows has at least one workflow
# - at least one workflow is called `main`
# - all tools exist in the db at the time of creation
# - input_schema is a valid json_schema
# - all tool call and yield steps refer to existing stuff

class Execution:
  id: Uuid
  created_at: datetime
  updated_at: datetime
  task_id: Uuid
  arguments: dict  # This is validated against task.input_schema
  status: Literal["queued", "starting", "running", "waiting-for-input", "success", "failed"]  # might need to reword these

class Transition:
  id: Uuid
  created_at: datetime
  execution_id: Uuid
  from_: tuple[CanonicalName, int]  # (workflow-name, step-index)
  to: None | tuple[CanonicalName, int]  # (workflow-name, step-index)
  output: dict
  type: Literal["finish", "waiting", "error", "step"]

Planned features

• sleep step
• multi-agent collaboration (multiple agents and multiple users in the same session)
• doc search step
• tool referenced by canonical name
• tasks of other agents can be tools
• (maybe) api calls
• (maybe) wait-for-input
• (maybe) else-if branches

Open questions

• semantics of the yield step, does it return back to the call origin or not?
• should we allow recursion?

[creatorrr]

is the language going to be yaml?

no, the api expects the input specification (the yaml here is just for readability) and we can add helper classes and utils in SDKs for creating them

[creatorrr]

Some additional ideas for the future:

tools_available:
- <system|agent-id>: <tool-name>
- <tool-name>
- <tool-uuid>
- name: send_message_to_guest
  partial:
    # this is partialed and removed from its signature before sending to the llm
    guest_id: inputs[0].guest_id

main:
-  map:
    arguments: ... # CEL that returns an array
    workflow: step1 # workflow name
    reduce: ... # workflow name (required)

[creatorrr]

example: when a pull request is merged, analyze the PR, write a song about it, convert it into audio and then upload to discord.

comments:

• we cant watch for PR merge event atm, that'd have to be done externally

name: make-a-song-about-it
description: ...

input_schema:
  type: object
  parameters:
    repo: # type string...
    pr_number: # type int
    discord_channel: # type int

# These tools need to be added to the agent
tools_available:
- fetch_pr
- generate_song_audio
- send_to_discord

# ideas
# 1. let's allow negative indices so inputs[-1] will refer to the previous input
# 2. let's make `$` an alias for outputs[-1]

main:
- tool: fetch_pr
  arguments:
    repo: $.repo
    id: $.pr_number
# let's say this returns {title: ..., description: ..., created_at, author, files_changed, comments, diffs}
# and we want the first 120 chars of each diff
- evaluate:
    first_chars: $.diffs.map(d, d.content[:120])
- prompt:
  - role: system
    content: |
      You are {{agent.name}}, {{agent.description}}. Here you are going to take a look at the following files and write a song about it:
      {% for file in outputs[-2].files_changed %}
        - {{file}}: {{$.first_chars[loop.index]}}
      {% endfor %}
      
      Start!
      
  settings:  # this is going to inherit the default_settings from the agent
    temperature: 1.0
- tool: generate_song_audio
  arguments:
    song: $.messages.choice[0].content
    voice: "ladygaga"
- tool: send_to_discord
  arguments:
    channel: inputs[0].discord_channel
    file:
      url: $.generated_audio.download_url
      

[whiterabbit1983]

Proposal #1

Can we use just an array for input arguments? for exmaple:

- tool: fetch_pr
  arguments:
    - repo
    - pr_number

Because it's gonna be something like arg1: $.arg1 all the time. I am not sure about the types, though

[whiterabbit1983]

How will tool calls be implemented? I'd like to see the rough PoC workflow