Skip to content
This repository has been archived by the owner on Feb 8, 2022. It is now read-only.

Pipeline data + capture #249

Merged
merged 11 commits into from
Oct 25, 2021
Merged

Pipeline data + capture #249

merged 11 commits into from
Oct 25, 2021

Conversation

sophiajt
Copy link
Contributor

@sophiajt sophiajt commented Oct 25, 2021

This moves us to a more thread-safe streaming while still maintaining as much performance as possible. To do this, we use a few tricks:

  • for declarations, we moved to using im currently. This should let us share the existing declarations more cheaply among the iteration steps without large amounts of copying or locking. This also lets us still access the declarations via id without an locking
  • moved stream out of value. Now we use PipelineData as the input and output of commands, an enum that is either a Stream or a Value
  • a modest amount of helper functions to ease the above transition
  • (hopefully) correct variable captures. This lets commands create the minimal scope necessary for running each block in a threadsafe way. There are copies involved, so it's not free, but performance for the tests I ran are similar to existing e-q performance
  • Split EvaluationContext into the EngineState and Stack components, as the latter needs to be passed mutably for the above. The end result is now every command has one additional parameter and we've simplified away the previous context structure.

TODO:

  • add external threading

@sophiajt sophiajt marked this pull request as ready for review October 25, 2021 23:03
@sophiajt sophiajt merged commit 301d1f6 into main Oct 25, 2021
@sophiajt sophiajt deleted the pipeline_data_capture branch October 25, 2021 23:04
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant