Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional specs #10094

Open
straight-shoota opened this issue Dec 18, 2020 · 0 comments
Open

Optional specs #10094

straight-shoota opened this issue Dec 18, 2020 · 0 comments

Comments

@straight-shoota
Copy link
Member

#10006 adds specs that only work when a suitable diff program is available. A similar situation exists in udp_socket_spec which has specs that are allowed to fail if multicast is not enabled (using pending! method).

This is great because it allows the spec suite to successfully complete even when it's run in an environment that lacks some features necessary for a spec to work.

There is however also a risk that this leniency leads to failing specs going unnoticed when the environment configuration is incomplete.
For example when the CI image for some reason doesn't have a diff program available, the respective specs would just not execute but everything report as good.

Typically, we want the main CI test jobs to run all specs and not skip any of them. And we make sure that the CI environment is properly configured for that. If some specs would fail the requirements to be executed, it should be considered a failure, not silently skip.

I propose to introduce a configuration for this, maybe an environment variable SPEC_ALLOW_SKIP. By default it's enabled. For spec runs that should not skip anything, it would be enabled. Then specs that raise Spec::PendingError would be considerd a failure.

Further enhancement could be named identifiers for spec groups that may be skipped, but that's overkill for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant