dora 0.3.5 Release

Better Error Logging and Graceful Stopping

Before

018ffe79-1c0c-7b68-8886-d8cff079c232
  2024-06-09T19:28:22.877741Z ERROR dora_daemon::spawn: 018ffe79-1c0c-7b68-8886-d8cff079c232/webcam: 
Traceback (most recent call last):
  File "/home/peter/Documents/work/dora/examples/python-dataflow/webcam.py", line 11, in <module>
    node = Node()
RuntimeError: failed to init event stream

Caused by:
    subscribe failed: Some nodes exited before subscribing to dora: {NodeId("plot")}

    This is typically happens when an initialization error occurs
                    in the node or operator. To check the output of the failed
                    nodes, run `dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 plot`.

Location:
    apis/rust/node/src/event_stream/mod.rs:90:17

    at binaries/daemon/src/spawn.rs:371


  2024-06-09T19:28:23.263843Z ERROR dora_daemon: 
    018ffe79-1c0c-7b68-8886-d8cff079c232/object_detection failed with exit code 1.

    Check logs using: dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 object_detection
                            
    at binaries/daemon/src/lib.rs:1077

  2024-06-09T19:28:23.264003Z ERROR dora_coordinator: some nodes failed:
                            
  - webcam: 
    018ffe79-1c0c-7b68-8886-d8cff079c232/webcam failed with exit code 1.

    Check logs using: dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 webcam
                            

Location:
    /home/peter/Documents/work/dora/binaries/coordinator/src/listener.rs:90:56
    at binaries/coordinator/src/lib.rs:279

dataflow failed: errors occurred in dataflow 018ffe79-1c0c-7b68-8886-d8cff079c232:
- machine ``:
    some nodes failed:
      - webcam: 
        018ffe79-1c0c-7b68-8886-d8cff079c232/webcam failed with exit code 1.
    
        Check logs using: dora logs 018ffe79-1c0c-7b68-8886-d8cff079c232 webcam
                                
    
    Location:
        /home/peter/Documents/work/dora/binaries/coordinator/src/listener.rs:90:56

After

01907807-9461-738c-a503-c92b82e0d4c2
attaching to dataflow (use `--detach` to run in background)
ERROR webcam: exited with code 1 with stderr output:
---------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/peter/Documents/work/dora/examples/python-dataflow/webcam.py", line 12, in <module>
    assert False, "could not start dataflow"
AssertionError: could not start dataflow
---------------------------------------------------------------------------------

INFO  plot: node finished successfully
INFO  object_detection: node finished successfully


[ERROR]
Dataflow 01907807-9461-738c-a503-c92b82e0d4c2 failed:

Node `webcam` failed: exited with code 1 with stderr output:
---------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/peter/Documents/work/dora/examples/python-dataflow/webcam.py", line 12, in <module>
    assert False, "could not start dataflow"
AssertionError: could not start dataflow
---------------------------------------------------------------------------------

Dynamic Node

We're introducing dynamic node, to run node in a native way, whether it is by calling python, jupyter or cargo. All you have to do is specify that the node is dynamic in the dataflow and specify the node id within the node code and you will be able to run the node natively without using dora start.

Getting Started

You can try minimal example:

cd examples/python-dataflow
dora start dataflow_dynamic.yml 
# In another terminal
python plot_dynamic.py 

• Then starting the dataflow

dora start dataflow.yaml

• Then using one of the following method to start the node:

Using interactive python:

$ python
>>> from dora import Node
>>> node = Node("node_0")
>>> event = node.next()
>>> event
{'id': 'tick', 'type': 'INPUT', 'value': <pyarrow.lib.NullArray object at 0x7bda86924460>
0 nulls, 'kind': 'dora', 'metadata': {'open_telemetry_context': ''}}

Using Python

$ python my_script.py # <-- node = Node("node_0")
# Log will appear in your terminal

Using Rust

$ cargo run my_node # <-- DoraNode::init_from_node_id(node_id);
# Log will appear in your terminal

Using a jupyter notebook

Made all dora python object representable in python

All dora-rs now implements: __str__, __repr__, __dir__, __dict__

Meaning that from Python, you will always be able to represent dora object as a Python Object.

>>> node
Node()
>>> event = node.next()
>>> event
{'id': 'tick', 'type': 'INPUT', 'value': <pyarrow.lib.NullArray object at 0x7bda86924460>
0 nulls, 'kind': 'dora', 'metadata': {'open_telemetry_context': ''}}
>>>
>>> from dora import Ros2NodeOptions, Ros2QosPolicies
>>>
>>> policies = Ros2QosPolicies()
>>> policies
Ros2QosPolicies(durability=Ros2Durability.Volatile, liveliness=Ros2Liveliness.Automatic, lease_duration=inf, reliable=false, max_blocking_time=0.0, keep_all=false, keep_last=1)
>>> policies.__dict__
{'keep_all': False, 'keep_last': 1, 'liveliness': Ros2Liveliness.Automatic, 'durability': Ros2Durability.Volatile, 'reliable': False, 'max_blocking_time': 0.0, 'lease_duration': inf}

Thanks a million to @EricLBuehler!

Simplified the dataflow

We removed one level of hierarchy by removing custom that was referencing custom node.

We also use the keyword path instead of source that might be a bit too confusing.

This change is backward compatible and you can keep the state of your previous dataflow.

Before:

  - id: idefics2
    custom:
      source: ../nodes/idefics2_node_demo.py
      inputs:
        text: whisper/text_llm
      outputs:
        - speak
        - control

After:

  - id: idefics2
    path: ../nodes/idefics2_node_demo.py
    inputs:
      text: whisper/text_llm
    outputs:
      - speak
      - control

Enabled dora on multiple machines

We can now experiment dora on different machine!

You can getting started with our multi-daemon example

A very minimalist setup would look like this

# On machine A
dora coordinator
dora daemon --machine-id A

# On machine B
dora daemon --machine-id B --coordinator-addr <COORDINATOR_IP>:53290

# On machine A or B
dora start ./examples/multiple-daemons/dataflow.yml

nodes:
  - id: rust-node
    _unstable_deploy:
      machine: A
    path: abs/path/to/target/debug/rust-node
    inputs:
      tick: dora/timer/millis/10
    outputs:
      - random

  - id: rust-sink
    _unstable_deploy:
      machine: B
    path: abs/path/to/target/debug/rust-sink
    inputs:
      message: rust-node/random

Thanks a million to @Gege-Wang and @XxChang!

I would also thanks @Michael-J-Ward and @LyonRust for their help on this release as well!!!

Full Changelog

• chore: Support RISCV64 by @LyonRust in #505
• Json schemas for VSCode YAML Support by @haixuanTao in #497
• Pretty Print Rust object when called from Python print by @haixuanTao in #503
• Fix Cargo.lock by @phil-opp in #506
• Use dependabot for automatic lockfile updates by @phil-opp in #507
• Run cargo update by @phil-opp in #508
• Allow top-level fields in node declaration by @phil-opp in #478
• Configure Renovate by @renovate in #509
• Make non-UTF8 stdout/stderr from nodes non-fatal by @phil-opp in #510
• Make dora cli connect to remote coordinator by @Gege-Wang in #513
• Provide help messages for CLI by @phil-opp in #519
• Renovate: group all dependency updates in single PR by @phil-opp in #524
• chore(deps): update dependencies by @renovate in #529
• Improve coordinator port config by @phil-opp in #520
• Fix some typos and add automatic typos check to CI by @EricLBuehler in #539
• Update Pyo3 bounds by @Michael-J-Ward in #472
• chore(deps): update dependencies by @renovate in #543
• Small logging improvements by @phil-opp in #537
• Refuse relative path for remote in coordinator by @XxChang in #538
• chore(deps): update rust crate clap to v4.5.7 by @renovate in #546
• Add --quiet flag to daemon and coordinator by @phil-opp in #548
• Implement file-based logging in daemon and coordinator by @phil-opp in #549
• Spawn daemon and coordinator in quiet mode on dora up by @phil-opp in #550
• Run dynamic node by @haixuanTao in #517
• Update dora new by @XxChang in #553
• fix event_as_input bug by @XxChang in #556
• Transform custom PyEvent into standard python dictionary for easi...

What's Changed

• Remove cxx_build call, which is no longer used by @phil-opp in #470
• Update ros2-client to latest version by @phil-opp in #457
• Configurable bind addrs by @Michael-J-Ward in #471
• Simple warning fixes by @Michael-J-Ward in #477
• Adding dora-rerun as a visualization tool by @haixuanTao in #479
• Fix Clippy and RERUN_MEMORY_LIMIT env variable default by @haixuanTao in #490
• Fix CI build errors by @phil-opp in #491
• Use resolver = 2 for in workspace in Rust template by @phil-opp in #492
• Add grace duration and kill process by @haixuanTao in #487
• Simplify parsing of AMENT_PREFIX_PATH by @haixuanTao in #489
• Convert rust example to node by @Michael-J-Ward in #494
• Adding python IDE typing by @haixuanTao in #493
• Fix: Wait until dora daemon is connected to coordinator on dora up by @phil-opp in #496
• Add dataflow_id getter in python by @haixuanTao in #501
• V0.3.4 by @haixuanTao in #499

New Contributors

• @Michael-J-Ward made their first contribution in #471

Full Changelog: v0.3.3...v0.3.4

What's Changed

• Remove cxx_build call, which is no longer used by @phil-opp in #470
• Update ros2-client to latest version by @phil-opp in #457
• Configurable bind addrs by @Michael-J-Ward in #471
• Simple warning fixes by @Michael-J-Ward in #477
• Adding dora-rerun as a visualization tool by @haixuanTao in #479
• Fix Clippy and RERUN_MEMORY_LIMIT env variable default by @haixuanTao in #490
• Fix CI build errors by @phil-opp in #491
• Use resolver = 2 for in workspace in Rust template by @phil-opp in #492
• Add grace duration and kill process by @haixuanTao in #487
• Simplify parsing of AMENT_PREFIX_PATH by @haixuanTao in #489
• Convert rust example to node by @Michael-J-Ward in #494
• Adding python IDE typing by @haixuanTao in #493
• Fix: Wait until dora daemon is connected to coordinator on dora up by @phil-opp in #496

New Contributors

• @Michael-J-Ward made their first contribution in #471

Full Changelog: v0.3.3...v0.3.4

What's Changed

• Metrics refactoring by @haixuanTao in #423
• Update bat dependency to v0.24 by @phil-opp in #424
• Add ROS2 bridge support for C++ nodes by @phil-opp in #425
• Provide function to create empty CombinedEvents stream by @phil-opp in #432
• Expose ROS2 constants in generated bindings (Rust and C++) by @phil-opp in #428
• Add option to send stdout as node/operator output by @haixuanTao in #388
• Fix warning about #pragma once in main file by @phil-opp in #433
• Send runs artefacts into a dedicated out folder by @haixuanTao in #429
• Create README.md for cxx-ros2-example by @bobd988 in #431
• Use Async Parquet Writer for dora-record by @haixuanTao in #434
• Update mio to fix security vulnerability by @phil-opp in #440
• Add initial support for calling ROS2 services from Rust nodes by @phil-opp in #439
• Enable ROS2 service calls from C++ nodes by @phil-opp in #441
• Use Debug formatting for eyre errors when returning to C++ by @phil-opp in #450
• Fix out-of-tree builds in cmake example by @phil-opp in #453
• Bump h2 from 0.3.24 to 0.3.26 by @dependabot in #456
• Llm example by @haixuanTao in #451
• Fix broken link in README by @mshr-h in #462
• fix cargo run --example cmake-dataflow compile bugs by @XxChang in #460
• Fix meter conflict by @haixuanTao in #461
• Update README.md by @bobd988 in #458
• Refactor README by @haixuanTao in #463
• Specify conda env for Python Operators by @haixuanTao in #468
• Release V0.3.3 by @haixuanTao in #465

New Contributors

• @bobd988 made their first contribution in #431
• @mshr-h made their first contribution in #462

Full Changelog: v0.3.2...v0.3.3