Recursive metadata application

[mkatychev]

Hello, I am trying to merge two records from two separate yaml sources:

# base.yaml
key1:
  value: ""
key2: true

and another variant to override it:

# force.yaml
key1:
  value: "non-empty"

---

# merge.ncl
let yaml_base =  import "base.yaml" in
let yaml_force =  import "force.yaml" in
{
  value | default = yaml_base,
} &
{
  value = yaml_force,
}

Running nickel export merge.ncl on the file above gives the output below::

{ value = { key1 = { value = "non-empty", }, }, }

Note that only record fields found in force.yaml are returned, this makes sense given the currency behaviour of & since the | default is applied to the value record field.

I would like to return:

{ value = { key1 = { value = "non-empty", }, key2 = true }, }

And do so by applying priorities recursively at runtime to the records found in base_yaml.

---

Since the SOT for the two records are found outside a nickel file, I cannot directly apply a priority to the fields without redundantly redefining each value, what would be the most ergonomic approach to give me the right recursive behaviour?

Ideally I'd like something akin to let yaml_base = (import "base.yaml" | default) in or any other way to recursively apply a priority.

Something akin to std.record.map_field_priority would be a great way to apply a merge priority in a functional manner:
base_yaml |> std.record.map_field_priority (fun _key _value => -1)

[mkatychev]

I am aware that 1.2 disabled recursive priorities with #1600, this is more of a discussion about setting a priority for a given field at runtime using a function such as std.record.map_field_priority that would not required a change to post RFC005 grammar.

[yannham]

So, if I understand correctly, you only want to set the priority of top-level fields in your YAML, not deeper than that?

In theory, there's no hard blocker for setting various metadata dynamically, like priority, contracts, or even documentation. However, philosophically, I fear this will lead to people implementing convoluted dynamic messy code that will build a value and its metadata entirely dynamically, while the whole idea of metadata was to have a static, immediately accessible information that can be leveraged by the language server and all the tooling easily. This breaks if we set those metadata at runtime.

That being said, I understand that you don't have a choice when importing existing data from a different format. So I'm not saying this is a no-go, but I want to be very careful of introducing such possibilities in a controlled way so that it doesn't become idiomatic.

[mkatychev]

Agreed, I am not advocating for a messy solution either nor opening avenues for it. Right now from a design perspective: to keep up with the philosophy there's a lot of upfront demand in keeping a nickel schema in sync with a YAML source of truth.

The current approach that I am taking for resolving recursive merging from imported records is to delete record fields that match a default "set" value such as "", {}, and [] by using std.record.filter.

I suppose if the source of truth was moved to nickel then this wouldn't be a problem.
An easier way to "scrub" imported records would also be helpful.

[yannham]

Right, I think it's fair to have something like that for this use-case. We just need to think about a design that makes it so that it doesn't get abused too much.

Tangentially, I know this might not be an option, but just in case, you can easily convert YAML to Nickel with nickel eval <<< 'import "somefile.yaml"' > somefile.ncl. The output of nickel eval should be valid Nickel.

[suimong]

Just thinking out loud here, would it be a possible solution if we take advantage of bidirectional transformation tool e.g. Augeas to annotate our metadata on certain nodes while keeping the SOC in the yaml/json file?

[mkatychev]

@yannham I think a fast way to generate a contract schema would go a long way into making it ergonomic. In a case where a development team is not fully bought into nickel the source of truth for a k8s deployment will stay in values.yaml for the time being. Having a way to generate/describe the (empty) object structure itself would be very helpful:

# base.yaml
key1:
  value: ""
key2: true

A hypothetical nickel describe base.yaml should produce the ncl below that can easily have | default or | forces added to it:

base.ncl
{ 
  key1 = { value, },
  key2, 
}

this way we can do something akin to

let base = (import "base.ncl") & (import "base.yaml") 
in
base & (import "force.yaml")