Use seed in CHOLMOD tests

[andreasnoack]

Uncontroversial. I just want to be sure that the tests pass on all platforms for the chosen seed.

As discussed in #11614 (comment)

[tkelman]

Thanks, looks like this'll work. Travis failure is unrelated.

[yuyichao]

I'm wondering seeding the random number generator can hide certain failures like #11658 and makes it harder to find those problems. (Just to be clear, I'm not saying that #11658 is easy either, it was fun watching hundreds of gdb sessions though....)

Or another way to do it (which I like more) is setup sth like Fuzz.jl on a buildbot to find these issues and leave the main testsuite clean and more deterministic.

[andreasnoack]

It would probably be a good idea if we had a single flag to turn the seeding on and off with the default to be on.

[StefanKarpinski]

There are two entirely different testing concerns:

• is this thing right in the first place – want to test lots of random cases, investigate when failures occur
• have we broken something that used to work – want to deterministically test things that we know work

The former requires human supervision and is generally interactive. It is also not what we're doing here. This is automated testing to make sure we haven't broken things. The seeds should be fixed.

[andreasnoack]

@StefanKarpinski The point is that the same tests can sometimes be used for both purposes. In that case it might be a good idea to have something like a flag instead of searching for srand statements to comment out in the test code.

[tkelman]

This seems like the kind of thing to add to the list of "stuff we would like a fancier test framework to handle for us."

[yuyichao]

Agree. It's on my todo list to get one for catching GC issues although I believe it would hit other bugs first...