feat(#8909): add CI jobs that run integration tests over a CHT launched in a K3D cluster

[dianabarsan]

Description

Adds two CI jobs to run integration tests over K3D architecture, using specific helm charts and local-path for persistent storage.

#8909

Code review checklist

• Readable: Concise, well named, follows the style guide, documented if necessary.
• Documented: Configuration and user documentation on cht-docs
• Tested: Unit and/or e2e where appropriate
• Internationalised: All user facing text
• Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.

Compose URLs

If Build CI hasn't passed, these may 404:

• Core
• CouchDB Single
• CouchDB Cluster

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[dianabarsan]

Hi @nydr . It would be great to get some feedback on this, if you have the bandwidth. Thanks a lot!

[nydr]

This is so cool! I was able to run the suite locally without much effort and I see a lot of potential here 😃

Am I correct in assuming a 30 minute extra build step is acceptable from a cost perspective?

I'm a bit concerned with the amount of potential false failures with flaky tests, if it becomes a big issue it might be worth considering skipping those by default in this suite.

Mostly minor comments, approving to unblock as I think it's good to merge as is. Overall impressive work!

[dianabarsan]

Hi @nydr
I addressed all comments and merged the templates into logical files.
In terms of context (re conversation the other day), it seems that k3d cluster create automatically switches default context to the new cluster. While I don't think it's an issue in CI, I'm wondering if we should try avoiding that for local runs, where the user might want to return to their previous context. Any ideas?

[nydr]

if we should try avoiding that for local runs, where the user might want to return to their previous context. Any ideas?

That would make sense, looks like it should be something like k3c cluster create <cluster> -kubeconfig-update-default=false and --kubeconfig-switch-context might need to be set to false too

[dianabarsan]

I pushed a bunch of changes trying to stabilize and unify how the two systems work together. @nydr would you mind having a look?

[nydr]

LGTM assuming the test failures are due to flaky tests

As a sidenote, would it make sense to tag all flaky tests and run those as a separate step to speed up retries and highlight the tests with issues? I think they're quite disruptive but out of scope for this PR

[nydr]

I like this format

[nydr]

Minor: I find this a bit confusing, I think it's ok to just describe what true does and then false is implied to be the inverse. Should it contain something like "always true for docker"? I generally don't like adding implementation details in docstrings, but I think it would be good to highlight that there's an override somehow

[dianabarsan]

I don't like this either.
The k3d container takes a longer time to start - compared to docker - and logs are cleared when the container starts. If I start watching logs before the container is up, in k3d I get an error. If I add a wait and retry, and the operation I'm waiting for happens immediately after container startup (this is the case), I could start watching the logs after the action had happened (this is also the case).

I'll add more info for the parameter.

[dianabarsan]

LGTM assuming the test failures are due to flaky tests

I don't believe they are just flakes. I'm fixing them. It's been a journey....