-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optional specs #10094
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
#10006 adds specs that only work when a suitable
diff
program is available. A similar situation exists in udp_socket_spec which has specs that are allowed to fail if multicast is not enabled (usingpending!
method).This is great because it allows the spec suite to successfully complete even when it's run in an environment that lacks some features necessary for a spec to work.
There is however also a risk that this leniency leads to failing specs going unnoticed when the environment configuration is incomplete.
For example when the CI image for some reason doesn't have a
diff
program available, the respective specs would just not execute but everything report as good.Typically, we want the main CI test jobs to run all specs and not skip any of them. And we make sure that the CI environment is properly configured for that. If some specs would fail the requirements to be executed, it should be considered a failure, not silently skip.
I propose to introduce a configuration for this, maybe an environment variable
SPEC_ALLOW_SKIP
. By default it's enabled. For spec runs that should not skip anything, it would be enabled. Then specs that raiseSpec::PendingError
would be considerd a failure.Further enhancement could be named identifiers for spec groups that may be skipped, but that's overkill for now.
The text was updated successfully, but these errors were encountered: