possible intermittent failure in lapack test

[JeffBezanson]

Here: https://travis-ci.org/JuliaLang/julia/jobs/288008723

The global RNG seed was 0x629c696f23fe04ca9e507ba916bcc18b.

Error in testset linalg/lapack:
Test Failed
  Expression: b ≈ c
   Evaluated: Float32[64.884, -83.3948, 53.8141, -30.9883, 37.1289, -43.2921, 59.0697, -21.2131, -25.1185, -37.884] ≈ Float32[64.9396, -83.4669, 53.8609, -31.0145, 37.1607, -43.3299, 59.1206, -21.2311, -25.1403, -37.9171]

[andreasnoack]

Those stack traces. It is not possible to see which test failed.

[StefanKarpinski]

@KristofferC, can we de-tersen the stack traces in some of these cases? Or is this lack of stack trace output not related to recent filtering changes?

[KristofferC]

This should have nothing to do with the stacktrace filtering. I can try it of course.

[KristofferC]

This is without filtering (by introducing a failure) when running make test-regex:

Test (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
regex: Test Failed
  Expression: [1, 2, 3] == [2, 3, 4]
   Evaluated: [1, 2, 3] == [2, 3, 4]
Stacktrace:
 [1] record(::Test.DefaultTestSet, ::Test.Fail) at /Users/kristoffer/julia/usr/share/julia/site/v0.7/Test/src/Test.jl:648
 [2] do_test(::Test.Returned, ::Expr) at /Users/kristoffer/julia/usr/share/julia/site/v0.7/Test/src/Test.jl:415
 [3] include_relative(::Module, ::String) at ./loading.jl:533
 [4] include at ./sysimg.jl:14 [inlined]
 [5] include(::String) at /Users/kristoffer/julia/test/testdefs.jl:13
 [6] macro expansion at /Users/kristoffer/julia/test/testdefs.jl:21 [inlined]
 [7] macro expansion at /Users/kristoffer/julia/usr/share/julia/site/v0.7/Test/src/Test.jl:940 [inlined]
 [8] macro expansion at ./util.jl:306 [inlined]
 [9] macro expansion at /Users/kristoffer/julia/test/testdefs.jl:19 [inlined]
 [10] anonymous at ./<missing>:?
 [11] eval(::Module, ::Any) at ./boot.jl:288
 [12] #runtests#48(::UInt128, ::Function, ::String, ::Bool) at /Users/kristoffer/julia/test/testdefs.jl:24
 [13] (::getfield(Main, Symbol("#kw##runtests")))(::Array{Any,1}, ::typeof(runtests), ::String, ::Bool) at ./<missing>:0
 [14] #145 at ./distributed/remotecall.jl:332 [inlined]
 [15] run_work_thunk(::getfield(Base.Distributed, Symbol("##145#146")){typeof(runtests),Tuple{String},Array{Any,1}}, ::Bool) at ./distributed/process_messages.jl:56
 [16] #remotecall_fetch#150(::Array{Any,1}, ::Function, ::Function, ::Base.Distributed.LocalProcess, ::String, ::Vararg{String,N} where N) at ./distributed/remotecall.jl:357
 [17] (::getfield(Base.Distributed, Symbol("#kw##remotecall_fetch")))(::Array{Any,1}, ::typeof(remotecall_fetch), ::Function, ::Base.Distributed.LocalProcess, ::String, ::Vararg{String,N} where N) at ./<missing>:0
 [18] #remotecall_fetch#154(::Array{Any,1}, ::Function, ::Function, ::Int64, ::String, ::Vararg{String,N} where N) at ./distributed/remotecall.jl:385
 [19] (::getfield(Base.Distributed, Symbol("#kw##remotecall_fetch")))(::Array{Any,1}, ::typeof(remotecall_fetch), ::Function, ::Int64, ::String, ::Vararg{String,N} where N) at ./<missing>:0
 [20] macro expansion at /Users/kristoffer/julia/test/runtests.jl:58 [inlined]
 [21] (::getfield(Main, Symbol("##42#46")))() at ./task.jl:335
Worker 1 failed running test regex:
Some tests did not pass: 35 passed, 1 failed, 0 errored, 0 broken.regex: Test Failed
  Expression: [1, 2, 3] == [2, 3, 4]
   Evaluated: [1, 2, 3] == [2, 3, 4]
Stacktrace:
 [1] record(::Test.DefaultTestSet, ::Test.Fail) at /Users/kristoffer/julia/usr/share/julia/site/v0.7/Test/src/Test.jl:648
 [2] (::getfield(Main, Symbol("##40#44")))() at /Users/kristoffer/julia/test/runtests.jl:165
 [3] cd(::getfield(Main, Symbol("##40#44")), ::String) at ./file.jl:70
 [4] include_relative(::Module, ::String) at ./loading.jl:533
 [5] include(::Module, ::String) at ./sysimg.jl:14
 [6] process_options(::Base.JLOptions) at ./client.jl:325
 [7] _start() at ./client.jl:391

Test Summary: | Pass  Fail  Total
  Overall     |   35     1     36
    regex     |   35     1     36
    FAILURE

After:

Test (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
regex: Test Failed
  Expression: [1, 2, 3] == [2, 3, 4]
   Evaluated: [1, 2, 3] == [2, 3, 4]
Stacktrace:
 [1] include_relative(::Module, ::String) at ./loading.jl:533
 [2] include at ./sysimg.jl:14 [inlined]
 [3] include(::String) at /Users/kristoffer/julia/test/testdefs.jl:13
 [4] macro expansion at /Users/kristoffer/julia/test/testdefs.jl:21 [inlined]
 [5] macro expansion at /Users/kristoffer/julia/usr/share/julia/site/v0.7/Test/src/Test.jl:937 [inlined]
 [6] macro expansion at ./util.jl:306 [inlined]
 [7] macro expansion at /Users/kristoffer/julia/test/testdefs.jl:19 [inlined]
 [8] anonymous at ./<missing>:?
Worker 1 failed running test regex:
Some tests did not pass: 35 passed, 1 failed, 0 errored, 0 broken.regex: Test Failed
  Expression: [1, 2, 3] == [2, 3, 4]
   Evaluated: [1, 2, 3] == [2, 3, 4]
Stacktrace:
 [1] record(::Test.DefaultTestSet, ::Test.Fail) at /Users/kristoffer/julia/usr/share/julia/site/v0.7/Test/src/Test.jl:645
 [2] (::getfield(Main, Symbol("##40#44")))() at /Users/kristoffer/julia/test/runtests.jl:165
 [3] cd(::getfield(Main, Symbol("##40#44")), ::String) at ./file.jl:70
 [4] include_relative(::Module, ::String) at ./loading.jl:533
 [5] include(::Module, ::String) at ./sysimg.jl:14
 [6] process_options(::Base.JLOptions) at ./client.jl:325
 [7] _start() at ./client.jl:391

Test Summary: | Pass  Fail  Total
  Overall     |   35     1     36
    regex     |   35     1     36
    FAILURE

A funny thing is that the file where the test failed (regex.jl)is not at all included in any of the stacktraces. I will try a full make testas well.

[KristofferC]

#24181 should show the exact location of a failed @test in all cases by storing the source location from the macro and not relying on any stacktraces.

[andreasnoack]

Would be great if this issue could be updated with a new test failure that includes line information.

[rfourquet]

Test Failed at /local/src/julia/test/linalg/lapack.jl:433

Awesome to now get the source location! @andreasnoack while you are on it, don't you think that these intermittent failures could be solved by using a more resilient test (like increasing atol? I'm not familiar with approximation testing...), instead of fixing a seed? the obvious advantage of not seeding is that the test is run with many different values over time and machines, increasing the confidence that what is tested is "true"; and if it came to fail somewhere, we now have the tool to reproduce the failure via --seed=SEED in runtests.jl.

[andreasnoack]

Thanks for the location of the test error. I'm not sure increasing the tolerance is better. For some these test failures, the error could be arbitrary since the probability that a randn(n,n) matrix is singular is non-zero (though small). I'd also prefer that the bound is no too swide too often so fixing the seed seems like the best solution. Regarding --seed=SEED, isn't there still a problem with reproducibility when testing with multiple processes?

[rfourquet]

Isn't it possible then to generate a random matrix until it's non-singular? Also I don't understand "the bound is no too side too often"... But anyway it was just a suggestion as I'm not used to do testing of a functionality with always the same input value, but this is your call. Regarding --seed=SEED, it should be reproducible whatever the number of processes, as srand(SEED) is called on each of them before running a test file.

[rfourquet]

Shall I fix this one with a fixed seed then?

[andreasnoack]

Also I don't understand "the bound is no too side too often"

Should have been wide not side

Regarding --seed=SEED, it should be reproducible whatever the number of processes, as srand(SEED) is called on each of them before running a test file.

👍

Shall I fix this one with a fixed seed then?

Yes, that would be great.