feat: Support for data sourcing and transformation with data template

[simster7]

Motivation

Users often source and transform data as part of their workflows. We want to add first-class support for these common operations in a new data template.

Goals

• Provide a varied set of sources and transformations so that users don't have to code and spin up their own Pods.

• Be able to "automagically" decide if we are able to do the transformations in-memory in the controller, thereby saving the need to create a new Pod.

• Provide an experience similar to common bash file manipulations. For example:

  cat file.txt | grep ... | sed ... | sort | uniq 
  

  Here, cat file.txt acts as a "source", but it could also be a curl or find, etc. grep ... | sed ... | sort | uniq act as transformations.

Maybe Goals

• Provide a quick and efficient way to do simple, individual, parameter-level transformations. For example:

  echo $(( $input + 1 ))
  

  However, this will likely be accomplished by an approach such as feat(controller): Expression template tags. Resolves #4548 & #1293 #5115

Non Goals

• Provide a substitute for a container. We want this to be a convenience for common use cases with sourcing and transforming data. If the convenience of the data template is not enough for our users, they should at said point define their own containers.

Examples

A data template could look like:

- name: generate-artifacts
  data:
    source:                       # Define a source for the data, only a single "source" is
                                  # permitted
      withArtifactPaths:          # Generate a list of all artifact paths in a given repository
        s3:
          bucket: test
          endpoint: minio:9000
          insecure: true
          accessKeySecret:
            name: my-minio-cred
            key: accesskey
          secretKeySecret:
            name: my-minio-cred
            key: secretkey
            
    transformation:               # The source is then passed to be transformed by
                                  # transformations defined here

      - filter:                   # Filters contents of source
          regex: "PROJECT_NAME"   # Only keep files that match this regex (e.g. contain
                                  # "PROJECT_NAME")
          
      - map:                      # Manipulate the files in some way
          replace:                # Self-explanatory
            old: "PROJECT_NAME"
            new: ""
            
      - group:                    # Group files, this can be very useful when used with withItems
          batch: 10               # Create groups of 10 files each. This can be useful to evenly
                                  # distribute work across pods expanded with withItems

data templates could also receive inputs from previous steps. In this case, since sourcing is not necessary, we can perform the transformations in-memory in the controller, saving the need to create a new pod and continuing immediately with execution.

- name: collect-artifacts
  inputs:
    parameters:
      - name: processed-file-list # Use and input parameter as source
  data:
    transformation:
      - filter:
          exclude: true           # Exclude if match
          regex: "\\.log"         # Don't include all log files generated by the processor
      - group:
          regex: ".+?\\.(.+?)$"   # Group by the result of the regex capture group, in this case
                                  # it's the file extension. This could be useful to post-process
                                  # or zip files differently based on extension

Open Questions

• How to provide extensive transformation without needing to enumerate them ourselves? Libraries such as expr and Sprig can be useful, but also limited (expr) or too domain specific (Sprig). Provide access to bash tools (e.g. grep, sed, etc.)? This is a balance between providing enough tooling for users to not need to spin up their own pods, but not enough that we're redoing work that users could do in their specific pods.

• What are the main use cases we should target with this template? What sources are a must-have? What transformations are a must-have?

• Could data: templates be used for quick in-line transformations? For example

  - name: collect-artifacts
    inputs:
      parameters:
        - name: processed
    data:
      transformation:
        - map:
            expression: "{{item}} + 1"

[simster7]

Compilation errors will be due to this change, which I have made to test my argoproj/pkg changes locally

[alexec]

I actually think we should return an error here (rather than panic) - and do the implementation later, in a v1.1

[alexec]

can I please ask you to remove this from this PR and we can follow up with >= v1.1

[simster7]

I actually think this is important this is shipped with the MVP as it protects from a possible exploit. Terry's comment describes it well: #4958 (review)

[terrytangyuan]

May I ask what v1.1 represents? I've been seeing that a lot in recent PRs.

[simster7]

v1.1 of this feature (not of Argo)

[alexec]

it's slang:

• v1.0 - the MVP
• v1.1/v1.2/v1.3 etc- an improvement to a feature we probably want follow-up soon after v1.0
• v2.0 - a complex/big improvement we want to hold off on on

[alexec]

give (a) this PR will only ship with artifactsPaths and (b) we only provide an expression transformation - I don't think we have a security risk with this PR?

[alexec]

regardless, lets narrow the scope v1.0 and follow up with a v1.1 after discussion

[alexec]

I've been thinking about other sources we might have. An obvious one is a normal artifact. This made me think about raw. Raw would get templated, which would be more powerful with expression tags (nee expression template, nee enhanced templates), but an expression or script would be more powerful always.

So I suggest:

• Descope `raw' from the PR (go narrower!!).
• Follow up with a new PR to introduce expression as a source, similar to raw you should be able to use inputs for this too.

I feel like we may end up with the following sources, each in its own PR based on community suggestions:

• artifact
• script
• raw

[simster7]

@alexec Done!

[alexec]

bump

[alexec]

still pending?

[alexec]

I think you could make this a testable example - BUT

a testable example should make a good example, if you need to be more complex, then better to have both examples and test

[simster7]

if you need to be more complex, then better to have both examples and test

I think in this case we need to be more complex. See comment: #4958 (comment)

[alexec]

200 lines of code - anyway we can reduce the maintenance this might need by making it somehow more focused?

[simster7]

It's 200 lines of node status, not code :) I have already reduced this as much as possible. (Including removing lines of node status that are not needed, such as Progress)

[alexec]

great - some mino comments which could be left to anther day

[alexec]

sure, we can change our mind later

[alexec]

I don't feel strongly, but it woc is MASSIVE and you might prefer to find your code in a single focussed file