refactor: reduce AttrScope usage

[ee7]

• Followed the steps in the contributor's guide: https://crashoverride.com/docs/other/contributing#filing-the-pull-request
• PR title uses semantic commit messages
• Filled out the template to a useful degree
• Updated CHANGELOG.md if necessary

The below line should enable slow tests for this PR:

tests:--slow

Issue

#269

Description

Continue (from 9c4fc0b) to prepare for the new con4m, by refactoring so that we get/set everything using the top-level chalk scope with fully-dotted keys. That helps prepare for the new con4m lookup interface, in con4m v2 was:

proc lookup*[T](ctx: RuntimeState, key: string): Option[T] =

where key is fully-dotted. And similarly in libcon4m:

extern c4m_value_t *
c4m_vm_attr_get(c4m_vmthread_t *tstate,
                c4m_str_t      *key,
                bool           *found);

Before this PR we were passing subscopes around, and my understanding was that in the new con4m:

• There's essentially a dictionary of fully-dotted keys, and therefore there's no longer the equivalent of AttrScope.
• We want to update the gets/sets to be fully string-based, matching the new design, as started all the way back in 66a5bb4.

Later we can try to:

• Add more compile-time safety again.
• Make the gets/sets more readable.

This PR also:

• Makes getChalkSubsections yield only key names.
• Replaces getObjectOpt with sectionExists.
• Makes getObject/getContents style more consistent.

---

I previously had about 20 more reviewable, self-contained and test-passing commits for this PR, but there were too many tricky conflicts with main, and I'd lost context on this part of the branch. So unfortunately it was best to squash before rebasing this on main, to reduce the time spent and bugs introduced when resolving conflicts.

With this PR:

$ git log -1 --format='%h %s'
ab6ed52 Merge branch 'main' into ee7/reduce-AttrScope
$ git grep --ignore-case --break --heading AttrScope
src/run_management.nim
20:proc getChalkScope*(): AttrScope =
27:    if v.isA(AttrScope):
30:proc sectionExists*(scope: AttrScope, s: string): bool =
41:proc con4mAttrSet*(attrs: AttrScope, fqn: string, value: Box, attrType: Con4mType) =

src/sinks.nim
177:proc getAuthConfigByName*(name: string, attr: AttrScope = AttrScope(nil)): Option[AuthConfig] =

For quick extra sanity checks, bad hacks for trying to find missing dots without parsing:

$ git diff -U0 main | grep '^[+]' | grep -o 'getChalkScope().*&.*)' | grep -v '.' | wc -l
0

$ git diff -U0 main | grep '^[+]' | grep -o 'getChalkScope().*&.*)' | cut -d',' -f2 | sort | uniq
 base & ".attempt_install")
 base & ".get_command_args")
 base & ".get_tool_location")
 base & ".produce_keys")
 content & ".use")
 getOutputConfig() & ".mark_template")
 getOutputConfig() & ".mark_template"))
 getOutputConfig() & ".report_template")
 "keyspec." & k & ".apply_substitutions")
 outconf & ".mark_template")
 outconf & ".report_template")
 "plugin." & plugin.name & ".codec")
 "plugin." & plugin.name & ".ignore")
 "plugin." & plugin.name & ".overrides")
 "plugin." & plugin.name & ".post_chalk_keys")
 "plugin." & plugin.name & ".post_chalk_keys"))
 "plugin." & plugin.name & ".post_run_keys")
 "plugin." & plugin.name & ".post_run_keys"))
 "plugin." & plugin.name & ".pre_chalk_keys")
 "plugin." & plugin.name & ".pre_chalk_keys"))
 "plugin." & plugin.name & ".pre_run_keys")
 "plugin." & plugin.name & ".pre_run_keys"))
 "plugin." & plugin.name & ".priority")
 "plugin." & p.name & ".ignore")
 "plugin." & p.name & ".overrides")
 report & ".report_template")
 report & ".use_when")
 scope & ".directories")
 scope & ".filepaths")
 scope & ".process_names")
 scope & ".strict")
 section & ".codec")
 section & ".conf_as_system")
 section & ".enabled")
 section & "." & k)
 section & "." & k).getOrElse("")
 section & "." & k).getOrElse(@[])
 section & "." & k).getOrElse(0)
 section & "." & k).getOrElse(int64(10 * 1048576))
 section & ".on_write_msg")
 section & ".pinned_cert_file"
 section & ".system")
 spec & ".enabled")
 spec & ".report_template")
 spec & ".sink_configs")
 spec & ".use_when")
 ss) and get[bool](getChalkScope()
 tmpl & ".doc")
 tmpl & ".key." & k & ".use")
 "tool." & info.name & ".stop_on_success")
 tplate & ".key." & k)
 tplate & ".key." & k & ".order")
 val & ".category")
 val & ".file_scope")
 val & ".file_scope.excluded_filepaths")
 val & ".file_scope.excluded_filetypes")
 val & ".file_scope.filepaths")
 val & ".file_scope.filetypes")
 val & ".file_scope.head")
 val & ".file_scope.regex"))
 val & ".host_scope")
 val & ".subcategory")
 v & ".callback")
 v & ".enabled")
 v & ".kind")
 v & ".kind") != int(kt)
 v & ".priority")
 v & ".value")

$ git diff -U0 main | grep -E '^[+][^+]' | grep -v 'getChalkScope' | grep -v '\.'
+                tplate: string): seq[string] =
+proc toJson*(dict: ChalkDict, tplate: string): string =
+                      t: string): string =
+  for k in getChalkSubsections("keyspec"):
+proc registerKeys(templ: string) =
+    for name in getChalkSubsections(section):
+        subscribedKeys[name] = true
+        if k notin hostInfo or
+  for name in getChalkSubsections("custom_report"):
+            override:
+        if k notin hostInfo or
+                     tmpl: string): Rope =
+      for k in getChalkSubsections("mark_template"):
+      for k in getChalkSubsections("report_template"):
+proc filterByTemplate*(dict: ChalkDict, p: string): ChalkDict =
+proc getOutputConfig*(): string =
+template getMarkTemplate*(): string =
+template getReportTemplate*(): string =
+    discard attrLookup(
+      ["key"],
+      ix = 0,
+      op = vlSecDef,
+    )
+    if item notin keys:
+    con4mAttrSet(
+      pack(true),
+      Con4mType(kind: TypeBool),
+    )
+      discard attrLookup(
+        ["key"],
+        ix = 0,
+        op = vlSecDef,
+      )
+      if item notin keys:
+      con4mAttrSet(
+        pack(true),
+        Con4mType(kind: TypeBool),
+      )
+  for k in getChalkSubsections("keyspec"):
+  for k in getChalkSubsections("tool"):
+    # Should have crashed by now if section does not exist :)
+  # tech stack rules
+  tsRules              = initHashSet[string]()
+proc hostHasTechStack(scope: string, proc_names: HashSet[string]): bool =
+    for langName in getChalkSubsections("linguist_language"):
+      languages[ext] = langName
+    for key in getChalkSubsections("tech_stack_rule"):
+  for key in getChalkSubsections("tech_stack_rule"):
+proc setPerChalkReports(tmpl: string) =
+template buildHostReport*(tmpl: string): string =
+  for topic in getChalkSubsections("custom_report"):
+iterator getChalkSubsections*(s: string): string =
+      yield k
+proc sectionExists*(scope: AttrScope, s: string): bool =
+  if not sectionExists(attrRoot, section):
+    error(section & " is referenced but its missing in the config")
+    error(section & " is misconfigured: " & getCurrentExceptionMsg())

[miki725]

overall looks fine but wonder if there is any AST thing we can use to test correctness?

[ee7]

Demonstrated that slow tests still pass here:

============ 178 passed, 1 skipped, 1 warning in 348.75s (0:05:48) =============

wonder if there is any AST thing we can use to test correctness?

It'd be possible. But a good fraction of problems here should be caught by the tests; for anything remaining, let's just try to shake out before the next release. Follow-up PRs will help too. Merging.