Add verbose option to @testset

[ronisbr]

This addresses a feature I proposed in #27088 (sorry for the delay 😊). The idea is to provide a mean to @testset to print child results even though all of them passed. IMHO (as I mentioned in the issue), the default behavior starts to become a problem when you have a very big package and wants to be sure that all necessary tests are working.

The way I coded is very simple: if verbose = true is passed to @testset, then the statistics of the children will be printed regardless of the test results. Notice that this does not affect the children of the children.

Thus, if we have:

using Test

@testset "Parent" verbose = true begin
    @testset "Child 1" verbose = true begin
        @testset "Child 1.1" begin
            @test 1 == 1
        end

        @testset "Child 1.2" begin
            @test 1 == 1
        end

        @testset "Child 1.3" begin
            @test 1 == 1
        end
    end

    @testset "Child 2" begin
        @testset "Child 2.1" begin
            @test 1 == 1
        end

        @testset "Child 2.2" begin
            @test 1 == 1
        end

        @testset "Child 2.3" begin
            @test 1 == 1
        end
    end

    @testset "Child 3" verbose = true begin
        @testset "Child 3.1" begin
            @test 1 == 1
        end

        @testset "Child 3.2" begin
            @test 1 == 1
        end

        @testset "Child 3.3" begin
            @test 1 == 1
        end
    end
end

we will get:

Test Summary: | Pass  Total
Parent        |    9      9
  Child 1     |    3      3
    Child 1.1 |    1      1
    Child 1.2 |    1      1
    Child 1.3 |    1      1
  Child 2     |    3      3
  Child 3     |    3      3
    Child 3.1 |    1      1
    Child 3.2 |    1      1
    Child 3.3 |    1      1

Notice that I chose to do not print the results of Child 2.

P.S.: In this PR, I added the functionality and the test so that we can discuss. After discussing, I can add the documentation as well.

[ronisbr]

Bump! :)

[gasagna]

I'd like to be able to specify the "depth of verbosity" with an option like verbose_depth and in some cases do not print anything at all, if verbose_depth=0.

[ronisbr]

Wouldn't it add unnecessary complications? I mean, with the current proposal, you can do that. Moreover, what would be the expected behavior if a parent has verbose_depth = 10 and the children verbose_depth = 1?

[ronisbr]

Bump # 2 :)

[ViralBShah]

Who would be the right person to review? @KristofferC ?

[ronisbr]

Looking at the commits to Test stdlib package, yes, I do think so.

[ronisbr]

Bump # 3 :)

[ronisbr]

Bump # 4

[StefanKarpinski]

Marking for triage so that this gets discussed.

[ronisbr]

Bump # 5 :) (please, tell me if this should be closed or if this is not the right time to implement such change)

[ronisbr]

Hi!

Is there any chance to consider this modification for Julia 1.6?

[vtjnash]

Needs a rebase, but sgtm

[ronisbr]

Nice! I will rebase then. Thanks for the feedback @vtjnash !

[ronisbr]

@vtjnash Done! The test failure in macOS does not seem related to this PR.