What is the minimum job config for the Evaluator?

[Etsija]

I'm creating an ORT Run and I am passing an empty configuration for the Evaluator, as the code comments in (jobConfigurations.kt) suggest that some configurations will have defaults. However, when running the Evaluator after successful Analyzer and Scanner jobs, I get these warnings to the logs:

ort-server-evaluator-worker-1  | 2024-05-27 06:24:28.893 [pool-1-thread-4] level=INFO  o.e.a.o.w.evaluator.EvaluatorRunner - No rule set path provided, using default path 'evaluator.rules.kts'.
ort-server-evaluator-worker-1  | 2024-05-27 06:24:28.906 [pool-1-thread-4] level=WARN  o.e.a.ortserver.config.ConfigManager - Could not read config file from default path 'copyright-garbage.yml'.
ort-server-evaluator-worker-1  | 2024-05-27 06:24:28.908 [pool-1-thread-4] level=WARN  o.e.a.ortserver.config.ConfigManager - Could not read config file from default path 'license-classifications.yml'.
ort-server-evaluator-worker-1  | 2024-05-27 06:24:29.147 [pool-1-thread-4] level=WARN  o.e.a.ortserver.config.ConfigManager - Could not read config file from default path 'resolutions.yml'.

The Evaluator seems to run OK, as its final status is FINISHED, but I get no results. This is from ort-result.yml:

evaluator:
  start_time: "2024-05-27T06:24:29Z"
  end_time: "2024-05-27T06:24:36Z"
  violations: []

The Reporter of course doesn't show any policy violations, either.

There probably should be some instructions as to the minimum job configuration needed for the Evaluator to run properly.

[mnonnenmacher]

The minimum configuration is indeed an empty object as the evaluator would then use all config files from their default locations and no package configuration provider.

Do you see any lines like this in the evaluator logs?
Evaluating rule '...' for package '...'.

The sample rule set for compose only has one rule that triggers if a package has no license at all, maybe that's just not the case for the repository you are scanning?

[Etsija]

Yes, I indeed have loads of these type of messages in the logs:

ort-server-evaluator-worker-1  | 2024-05-27 06:24:35.974 [pool-1-thread-4] level=INFO  org.ossreviewtoolkit.evaluator.Rule - Evaluating rule 'NO_LICENSE_IN_DEPENDENCY' for package 'NPM::fs-constants:1.0.0'.
ort-server-evaluator-worker-1  | 2024-05-27 06:24:35.974 [pool-1-thread-4] level=INFO  org.ossreviewtoolkit.evaluator.Rule -    !(isProject()) == true
ort-server-evaluator-worker-1  | 2024-05-27 06:24:35.974 [pool-1-thread-4] level=INFO  org.ossreviewtoolkit.evaluator.Rule -    !(isExcluded()) == false
ort-server-evaluator-worker-1  | 2024-05-27 06:24:35.974 [pool-1-thread-4] level=INFO  org.ossreviewtoolkit.evaluator.Rule -    Rule skipped.

So I guess the sample ruleset is so minimal that it doesn't actually trigger any policy violations. We (Double Open Oy) are using something like this, so I guess the default ruleset could be extended also for Docker Compose setup, if not else than for demonstration purposes, and for the purpose of proving the concept, what do you think?

[Etsija]

Actually, thinking of this, is there any reason why this couldn't be used instead, as ORT also uses that as an example of a rule file?

[sschuberth]

Actually, thinking of this, is there any reason why this couldn't be used instead, as ORT also uses that as an example of a rule file?

We have a long-standing discussion about the purpose / scope of examples rules for ORT. Basically, it boils down to the question whether examples should be a "reasonable default implementation", or an "synthetic implementation to show the features" only.

Currently, both

https://github.com/oss-review-toolkit/ort/blob/main/examples/example.rules.kts

and

https://github.com/oss-review-toolkit/ort-config/blob/main/evaluator.rules.kts

are a mix of both.

Personally, I'd strongly vote for at least the former to be a "synthetic implementation" that triggers dummy rule violations for any code, just to show the concept, and not make people wonder why nothing gets triggered, as happened in the scope of this issue.

[Etsija]

Yes, I agree there should be a rules file which triggers at least some violations, albeit not 100% correctly from the standpoint of someone curating a project, as with the current rules file, I have been wondering for a few days already why the Evaluator is not triggering anything, as if it isn't even run, although run results show it as FINISHED. So I just thought I've been mis-configuring the job.

[mnonnenmacher]

Personally, I'd strongly vote for at least the former to be a "synthetic implementation" that triggers dummy rule violations for any code, just to show the concept, and not make people wonder why nothing gets triggered, as happened in the scope of this issue.

I agree with this, the file is also rendered here and a shorter example that showcases the features would be more helpful.

For this repository I would like to add simple examples for the other configuration files as well. For example, a license classifications file which classifies only few licenses. Then another rule could be added that triggers if a non-classified license is found, which would be very likely for most projects.

[sschuberth]

@Etsija has this been answered? If so, please close.

[Etsija]

The original question, yes, but the discussion took another turn and implied that some files were to be added to scripts/compose/config and I wanted to keep this open. But best to close it and maybe open a new one later if needed.