-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some flaky tests #1329
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
We tried running your project and discovered that it contains some flaky tests (i.e., tests that nondeterministically pass and fail). We found these tests to fail more frequently when running them on certain machines of ours.
To prevent others from running this project and its tests in machines that may result in flaky tests, we suggest adding information to the README.md file indicating the minimum resource configuration for running the tests of this project as to prevent observation of test flakiness.
If we run this project in a machine with 1cpu and 1gb ram, we observe flaky tests. We found that the tests in this project did not have any flaky tests when we ran it on machines with 2cpu and 4gb ram.
Here is a list of the tests we have identified and their likelihood of failure on a system with less than the recommended 2 CPUs and 4 GB RAM.
Reproducing
Build the image:
Running:
this configuration likely prevents flakiness (no flakiness in 10 runs)
checking results
this other configuration –similar to the previous– can’t prevent flaky tests (observation in 10 runs)
Thank you for your attention to this matter. We hope that our recommendations will be helpful in improving the quality and performance of your project.
The text was updated successfully, but these errors were encountered: