Import arbitrarty YAML/JSON

[smyrman]

Is your feature request related to a problem? Please describe.

Sometimes it would be nice to be able to import arbitrary data from other YAML (or JSON) files, and use values from these files without having to duplicate configuration.

One concrete example that we will look at below, is how to use a config files generated by appctl. This file looks something like this:

cluster_name: <name>
project: <project>
compute:  # choose "region" or "zone"
   region: <region>
namespace: <namespace>
no_pr_approval_required: false

It would be nice to be able to extrat and work with these variables in tusk in one way or another.

Describe the solution you'd like

It's not important to me how this works. But here is one possible syntax:

imports: # new section imports
   <name>: # name of import
     path: <file> # path to file to import
     expose: # map of values to expose, and what to expose them as.
       <local name/alias>: <dotted path in file>

Full example:

imports: # new section imports
  delivery-dev:
    path: delivery/envs/dev.yaml
    expose: &delivery-expose # Using a YAML anchor to keep things DRY
      cluster: cluster_name
      project: project
      region: compute.region
      namespace: namespace
 delivery-staging: 
    path: delivery/envs/staging.yaml
    expose: *delivery-expose
  delivery-prod:
    path: delivery/envs/prod.yaml
    expose: *delivery-expose
    
tasks:  
  use-context:
    usage: "Switch to the cluster belonging to a particular deployment"

    args:
      env:
        usage: Name of environment
        values:
          - dev
          - staging
          - prod

    run:
      - command: kubectl config use-context gke_${delivery.project}_${delivery.region}_${delivery.cluster}
         with-alias: # another feature to alias variables (not strictly needed for this, just reduce code).
           delivery: "imports.delivery-${env}"

The final run line without an "alias" feature:

    run:
      - command: kubectl config use-context gke_${imports.delivery-dev.project}_${imports.delivery-dev.region}_${imports.delivery-dev.cluster}
         when:
         - {"equal":  {"env": "dev"}
      - command: kubectl config use-context gke_${imports.delivery-staging.project}_${imports.delivery-staging.region}_${imports.delivery-staging.cluster}
         when:
         - {"equal":  {"env": "staging"}
      - command: kubectl config use-context gke_${imports.delivery-prod.project}_${imports.delivery-prod.region}_${imports.delivery-prod.cluster}
         when:
         - {"equal":  {"env": "prod"}

The purpose of this syntax example is not to propose an actual syntax, but to explain the feature.

Additional context

Possible concern with example syntax:

• As options are not scoped in ${} clauses, an option named "imports" would conflict with the imports section.

Possible solutions:

• Make imports part of options.
• Include imports in same namespace as options and set a priority; e.g. global option over imports, local options over global options. I.e. we refer to "${delivery-dev}" not "${imports.delivery-dev}".

[moondev]

Another solution for this is simply including the yaml as additional documents in tusk.yaml and reading them with a utility like yq https://github.com/mikefarah/yq

tusk only reads the first document and ignores additional yaml documents inside tusk.yaml

note: golang 1.17.x required for this tusk.yaml example

#!/usr/bin/env -S go run github.com/rliebz/[email protected] -f
---
tasks:
  read-one:
    run:
      - go run github.com/mikefarah/yq/v3@latest r tusk.yaml -d1
  read-two:
    run:
      - go run github.com/mikefarah/yq/v3@latest r tusk.yaml -d2 spec

---
document: one

---
spec:
  document:
    two: example

./tusk.yaml read-one
./tusk.yaml read-two

[smyrman]

Cool trick @moondev.