Send runs artefacts into a dedicated out folder
#429
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phil-opp
approved these changes
Feb 28, 2024
There was a problem hiding this comment.
Good idea, thanks!
As the tree structure is based on the dataflow_id which is random, the dataflow folder order is not ordered from oldest to latest dataflow which can be confusing.
I can prefix the dataflow folder with a timestamp.
We could switch from v4 UUIDs to v7 UUIDs, which start with a UNIX timestamp. This way, they should be sortable.
…r to make it easier to manage multiple data coming from multiple runs of a dataflow
…_dir` to make it easier to check
phil-opp
approved these changes
Feb 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will send runs artefacts into a
outfolder instead of the previously used OStemp_dir. This makes it easier to check log files as well as make it clearer to segment the logs per runs of a same dataflow.Exemple of the tree structure of the out folder
logs_input.arrowwithout prefixCurrent Limitation
As the tree structure is based on the dataflow_id which is random, the dataflow folder order is not ordered from oldest to latest dataflow which can be confusing.
I can prefix the dataflow folder with a timestamp.