Document & expose printing of custom AbstractTestSet

[Seelengrab]

This was previously alluded to in the docs as being possible ("just record if we're not the top-level parent"), but trying to do that didn't actually do anything because the printing in the DefaultTestSet ignored anything that wasn't a Test.Result or Test.DefaultTestSet. This is particularly problematic if any of the child-testsets had failures, because those failures would never be shown to a user.

An example implementation, with & without hooking into showing the gathered counts

        using Test

        mutable struct CustomPrintingTestSet <: Test.AbstractTestSet
            description::String
            passes::Int
            fails::Int
            errors::Int
            broken::Int
        end
        CustomPrintingTestSet(desc::String) = CustomPrintingTestSet(desc, 0,0,0,0)

        Test.record(cpts::CustomPrintingTestSet, ::Test.Pass) = cpts.passes += 1
        Test.record(cpts::CustomPrintingTestSet, ::Test.Error) = cpts.errors += 1
        Test.record(cpts::CustomPrintingTestSet, ::Test.Fail) = cpts.fails += 1
        Test.record(cpts::CustomPrintingTestSet, ::Test.Broken) = cpts.broken += 1
        Test.get_test_counts(ts::CustomPrintingTestSet) = Test.TestCounts(
                                                                true,
                                                                ts.passes,
                                                                ts.fails,
                                                                ts.errors,
                                                                ts.broken,
                                                                0,
                                                                0,
                                                                0,
                                                                0,
                                                                Test.format_duration(ts))

        function Test.finish(cpts::CustomPrintingTestSet)
            if Test.get_testset_depth() != 0
                Test.record(Test.get_testset(), cpts)
                # printing is handled by the parent
                return cpts
            end

            Test.print_test_results(cpts)
            cpts
        end

        struct NonRecordingTestSet <: Test.AbstractTestSet
            description::String
        end
        Test.record(nrts::NonRecordingTestSet, ::Test.Result) = nrts
        Test.finish(nrts::NonRecordingTestSet) = Test.record(Test.get_testset(), nrts)

         @testset "outer" begin
            @testset "a" begin
                @test true
            end
            @testset CustomPrintingTestSet "custom" begin
                @test false
                @test true
                @test_broken false
                @test error()
            end
            @testset NonRecordingTestSet "no-record" begin
                @test false
                @test true
                @test_broken false
                @test error()
            end
            @testset "b" begin
                @test true
            end
        end

Before:

After:

[Seelengrab]

Looks like there was a segfault while running the GC tests? Seems unrelated to this PR though.

[Seelengrab]

I had hoped that git would be smart enough to do the conflict resolution with #53212 on its own, but oh well - rebased onto master, so with a bit of luck the GC tests won't segfault this time around.

[IanButterworth]

Seems like the right thing to do to me.

I wonder whether this will affect the test harness stuff in vscode @davidanthoff @pfitzseb

[Seelengrab]

I wonder whether this will affect the test harness stuff in vscode

The only thing I could find in the julia-vscode org was this:

https://github.com/julia-vscode/TestItemRunner2.jl/blob/54d3970a949b0e6b6511d3de2398c4d7b1077adc/packages/JuliaInterpreter/test/juliatests.jl#L155

Whether that is currently used by the extension, I don't know. Either way, this function was previously internal & only defined for DefaultTestSet, while the code in TestItemRunner2.jl assumes it works for any AbstractTestSet. The thing that would have to change there is remove the iteration-destructuring, which would make the code (IMO) easier to read, just like it's now easier to follow the code in Base.

[pfitzseb]

Yeah, I don't think a potential breakage in the extension should hold this up. We can always fix that in time for 1.11.

[Seelengrab]

Nice, thanks everyone for the quick turnaround on this! You've just unblocked some work I'm hoping to release relatively soon :)