Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI (Buildbot): temporarily skip almost all of the tests on FreeBSD if we are on CI and the Julia version is 1.8+ #42951

Closed
wants to merge 3 commits into from

Conversation

DilumAluthge
Copy link
Member

@DilumAluthge DilumAluthge commented Nov 5, 2021

Because the tester_freebsd64 Buildbot job is always failing, every single commit on master has an ❌, which is really noisy and unhelpful.

Until #41955 is ready, let's temporarily skip almost all of the tests if all three of the following conditions hold:

  1. We are on FreeBSD.
  2. We are running on CI, as evidenced by the value of the CI environment variable.
  3. The Julia version is 1.8+. (FreeBSD tests are only broken on 1.8+. We expect the FreeBSD tests to be passing on 1.6 and 1.7.)

If all three of those conditions hold, we will skip almost all of the tests. That is, we will run a few test sets, just as a sanity check, But we will skip all other test sets.

This is just a short-term temporary solution. When #41955 is ready to go, we will revert this change.

@DilumAluthge DilumAluthge added the domain:ci Continuous integration label Nov 5, 2021
@DilumAluthge DilumAluthge added the system:freebsd Affects only FreeBSD label Nov 5, 2021
@DilumAluthge DilumAluthge marked this pull request as draft November 5, 2021 10:09
@DilumAluthge DilumAluthge marked this pull request as ready for review November 5, 2021 10:17
@DilumAluthge
Copy link
Member Author

Note that tester_freebsd64 is still going to fail on this PR, but that is because of #42944

@DilumAluthge DilumAluthge mentioned this pull request Nov 5, 2021
@ararslan
Copy link
Member

ararslan commented Nov 5, 2021

My primary concern here is the potential to regress further. If we skipped FreeBSD on master, we wouldn't have caught #42944.

@ViralBShah
Copy link
Member

Isn't there a way to keep freebsd64 running, but have it not contribute to the overall green?

@DilumAluthge
Copy link
Member Author

My primary concern here is the potential to regress further. If we skipped FreeBSD on master, we wouldn't have caught #42944.

So, we're not actually skipping all of the tests. We still run a few test sets, which should be sufficient to catch bugs like #42944.

Actually, as an example, look at the tester_freebsd64 job on this PR. If you look at the log, the job still fails due to #42944, which means that we would still have caught #42944.

@DilumAluthge
Copy link
Member Author

Isn't there a way to keep freebsd64 running, but have it not contribute to the overall green?

With Buildkite, yes. For example, we are already doing that with the tester_musl64 Buildkite job. We always run the tester_musl64 job, but it is marked as an "allowed failure", which means that even if it fails, the overall Buildkite job will still be green.

With Buildbot, I don't know if that is possible.

@DilumAluthge DilumAluthge changed the title CI (Buildbot): temporarily skip all of the tests on FreeBSD if we are on CI and the Julia version is 1.8+ CI (Buildbot): temporarily skip almost all of the tests on FreeBSD if we are on CI and the Julia version is 1.8+ Nov 5, 2021
@ararslan
Copy link
Member

ararslan commented Nov 6, 2021

I am actually quite close to a fix, since with Jameson's help the upstream issue with libunwind has been identified and I have an upstream PR, so this PR shouldn't be necessary after all.

@DilumAluthge DilumAluthge marked this pull request as draft November 6, 2021 17:11
@DilumAluthge DilumAluthge deleted the dpa/skip-all-tests-freebsd branch November 6, 2021 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain:ci Continuous integration system:freebsd Affects only FreeBSD
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants