#13385 [Clojure] - Turn examples into integration tests

[hellonico]

!!! This is work in progress.

Would:

• the committed script
• adding a bare minimum deftest inside each example

be ok?

[gigasquid]

Thanks so much @hellonico for tackling this. This will be a big improvement to stability and regression of the package as we develop it. The way you are approaching it with a minimum test and then the script seems like the right direction. 😸

[hellonico]

The following examples are turned into integration tests:

SuperPinkicious:examples niko$ find . -name test
./gan/test
./visualization/test
./neural-style/test
./tutorial/test
./multi-label/test
./module/test
./cnn-text-classification/test
./imclassification/test
./profiler/test
./rnn/test

rnn has a training section and a test trained network section, but the training section is very, very, very slow
I am wondering if it should still be added or not? It largely depends on the CI setup at this stage, but on my dev machine I can not get anything in a reasonable amount of time.

[gigasquid]

@hellonico - Awesome work!
Regarding the rnn example - I think limiting it to just the test trained network is sufficient and will keep the time down to a somewhat reasonable level. I would like to be able to run the suite locally as well as CI.

[hellonico]

Good morning !
Great. Just squashed the commits ... and rebased on master.

[gigasquid]

Great! I’ll take a look on Monday. Thanks! Have a good rest of the weekend :) Carin

…

On Sat, Dec 8, 2018 at 9:39 PM Nicolas Modrzyk ***@***.***> wrote: Good morning ! Great. Just squashed the commits ... and rebased on master. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#13554 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUxS-canYs9mrGwrz-E-AJKtb0M5D6Uks5u3HfwgaJpZM4ZFsnX> .

[hellonico]

ci/jenkins/mxnet-validation/sanity — Job failed

Any idea on how to avoid the (de facto not very possible) license check on json files ?

[hellonico]

Checked on OSX, and on two linux boxes:
manjaro.txt
ubuntu.txt

Finally ...

[gigasquid]

Awesome work! I think we can get the CI to pass the RAT check if you put the files having trouble:

mxnet/contrib/clojure-package/examples/imclassification/test/test-symbol.json.ref
mxnet/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref

in here https://github.com/apache/incubator-mxnet/blob/master/tests/nightly/apache_rat_license_check/rat-excludes

It gets called from here https://github.com/apache/incubator-mxnet/blob/master/ci/docker/runtime_functions.sh#L996

@marcoabreu Do you have thoughts on how these integration tests should be integrated in the CI? Should it be nightly since they run pretty long? Should we merge this first and then open another one for integrating into the CI?

[marcoabreu]

Yeah nightly would be a good place. I'll leave it up to you whether we do it in this PR or a separate one. But I think a separate one is a good idea

[hellonico]

• Rebased master (monday 10th commits)
• Updated json files to be ignored in the nightly checks

Ready to be merged !

Let's find out how much time it takes to run the suite. When the models are already there, those tests run in about 10-15 minutes on a slow machine.

[gigasquid]

Thanks for taking this on and making it happen 💯

I'm going to go ahead a merge this and then we can work on adding it to the nightly tests in another PR and see how it runs there.