ci: Daily CI to soak flaky tests #5382
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Setting up a daily scheduled build to run the LSP tests 5 times on each platform, in an effort to proactively uncover flaky test failures that show up at the worst possible time as you're trying to get your wonderful PR you've been working on for weeks merged or even worse that break the nightly build for the third day in a row forcing the team to drop everything to click Retry Failed Jobs two or three times before ANY PRs can be merged not that I'm frustrated or anything. :)
The PR CI was previously running the LSP tests twice every time for the same reason. Given this daily job will soak them more deeply, and that the
osxunit test job on PRs in particular has been taking almost 45 minutes and becoming the limiting factor, I've opted to just run them once on PRs now.Note that unlike the original double run, it was simpler to use matrices just as we do for integration tests, only in this case just iterating the same test run in parallel 5 times. It's possible running twice on the same runner might trigger more failures, but it seems unlikely since the second run happens in a fresh process and no state leaks between the runs AFAICT.
How has this been tested?
Dry-run on my fork: https://github.com/robin-aws/dafny/actions/runs/8914692810
I've run it a few times without any failures yet, but we'll have to see what happens after a week or so of scheduled runs.
By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.