So you propose an entire new stat for this? Success, failed, errored, pending, skipped?

I doubt it's much use to report skipped and pending specs separately in the spec output. They should probably just be listed combined as skipped, whether it's because the spec is not implemented (pending) or for a different reason (skip!).

@straight-shoota Due to the different semantics there is a room for usefulness in reporting them separately.

Sure, there's room for later improvement ;) But right now, I'd like to simply change to skip and nothing more. pending! hasn't been in a release yet, so we can easily change it without reprecation.
Currently there isn't much use of it anyway, so let's have it in a release first and at a later point we can see if the reporting needs more granular refinement or not. There's nothing wrong with summarizing pending and skip! in the spec output.

I don't see much point in renaming it. Any spec that's not run is "awaiting conclusion" in some way, otherwise it would not have been written in the first place. In the multicast example that conclusion just lies within the external environment rather than the spec itself.

But then I also cannot really care less about how the functionality is called. I just didn't want to introduce yet another concept.

But these really are different concepts. Whether a spec is not implemented or can't run may ask for different handling. See #10094 for how this matters.

I agree with @jhass. This would create a new concept and I see no need for that.

My choice to reduce the chance of a silent failure in the CI would be to add a "max pending allowed" or "error on pending". If that is not enough, allow "error on pending!" only could do the trick: like SPEC_ERROR_ON_PENDING=YES|BANG|NO.

Having something that an skipped spec, that is not pending is confusing to me.

In RSpec, one of the differences with pending is that you can actually implement the test and let it execute, but its failure condition is reversed. This allows you to know that once the pending condition is no longer valid, you can un-mark it as pending.

https://rspec.info/documentation/3.9/rspec-core/RSpec/Core/Pending.html

#pending
Marks an example as pending. The rest of the example will still be executed, and if it passes the example will fail to indicate that the pending can be removed.

#skip
Marks an example as pending and skips execution.

@bcardiff "error on pending!" does not make any sense whatsoever with the pre-existing pending method. This is a static designation for specs that are not expected to worth or not even implemented. Having such a pending example in the code base should never be a reason to fail the spec suite.

But pending! is different: It is used to skip specs that are normally expected to work but can't execute because of runtime contraints. Whether skipping an example should fail the spec suite depends on the environment. In an environment that doesn't support multicast, it's okay to skip, in an environment that is expected to support multicast it is not okay.

That is already a different concept. I'm not suggesting to introduce it, it's already here since #9566. This proposal is just about calling it out by not using the same names pending and pending! which confuses those different concepts.