Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: store strings in comparison test results instead of the original objects #24847

Merged
merged 1 commit into from
Nov 30, 2017

Conversation

JeffBezanson
Copy link
Sponsor Member

This avoids deserialization errors; might fix #20230.

Motivated by a nasty one of these I got recently: https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.21696/job/ernf3v0vhorc5w0f

@JeffBezanson JeffBezanson added the testsystem The unit testing framework and Test stdlib label Nov 29, 2017
@JeffBezanson
Copy link
Sponsor Member Author

Ok, unsurprisingly it takes a huge amount of time to stringify every test argument, so now this only stringifies when the test fails.

This also uncovered an obscure issue in show_default where forming a pair containing a shown value could fail if the value has bad convert methods.

@JeffBezanson JeffBezanson merged commit 016b23c into master Nov 30, 2017
@JeffBezanson JeffBezanson deleted the jb/Test_repr branch November 30, 2017 22:17
@ssfrr
Copy link
Contributor

ssfrr commented Jul 11, 2018

This broke TestSetExtensions.jl, which relies on having access to the LHS and RHS of the comparison so it can display a pretty-printed diff on test failures.

Is it possible for this stringification to happen after the result is passed to the TestSet, so that custom TestSets can still have access to the values?

As a workaround I can eval the LHS and RHS of the original expression, but that's bad if they have side-effects.

@ssfrr
Copy link
Contributor

ssfrr commented Jul 12, 2018

hrm, eval isn't a great workround also because I'm evalling in the context of my code so it can't access any surrounding variables, e.g.

julia> x = [3, 4, 5];

julia> @testset ExtendedTestSet "testing" begin
           @test x == [1, 2, 3]
       end

I also don't understand the serialization problem that this PR was attempting to address well enough to have a sense for whether this could be easily reverted.

@iamed2
Copy link
Contributor

iamed2 commented Jul 12, 2018

Breaking TestSetExtensions in this way would be very unfortunate

@ssfrr
Copy link
Contributor

ssfrr commented Aug 7, 2018

Is there any chance this would be reverted? I'm thinking another option would be to just cut-paste Base.Test into TestSetExtensions rather than extending it, but that seems like a shame.

@ssfrr
Copy link
Contributor

ssfrr commented Aug 17, 2018

I talked to @JeffBezanson on slack about this. It seems like the thing to do is to move the stringification further down the chain, so they are still accessible to the testset, but get stringified before they make it to a different machine. I don't have time to get to it immediately, but hopefully within a couple weeks (if no one gets to it first...).

@ssfrr
Copy link
Contributor

ssfrr commented Aug 17, 2018

Oh, there was also some conversation about whether this would be a breaking change, but I'm pretty sure that accessing the data field of the result is considered mucking about with internals (AFAICT it's not documented anywhere), so it seemed like there was consensus that it's not considered breaking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testsystem The unit testing framework and Test stdlib
Projects
None yet
Development

Successfully merging this pull request may close these issues.

testsystem: original error lost if a worker response results in a deserialization error on node 1
3 participants