Flaky test test_io:test_ImageRecordIter_seed_augmentation

[marcoabreu]

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11340/8/pipeline

test_io.test_ImageRecordIter_seed_augmentation ... [21:50:02] src/io/iter_image_recordio_2.cc:170: ImageRecordIOParser2: data/cifar/train.rec, use 4 threads for decoding..
[21:50:02] src/io/iter_image_recordio_2.cc:228: Load mean image from data/cifar/cifar10_mean.bin
[21:50:02] src/io/iter_image_recordio_2.cc:246: Load mean image from data/cifar/cifar10_mean.bin completed
[21:50:02] src/io/iter_image_recordio_2.cc:170: ImageRecordIOParser2: data/cifar/train.rec, use 4 threads for decoding..
[21:50:02] src/io/iter_image_recordio_2.cc:228: Load mean image from data/cifar/cifar10_mean.bin
[21:50:02] src/io/iter_image_recordio_2.cc:246: Load mean image from data/cifar/cifar10_mean.bin completed
FAIL


======================================================================
FAIL: test_io.test_ImageRecordIter_seed_augmentation
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/work/mxnet/tests/python/unittest/test_io.py", line 381, in test_ImageRecordIter_seed_augmentation
    assert(np.array_equal(data,data2))
AssertionError

Probably due to missing tolerances

[marcoabreu]

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1042/pipeline

[marcoabreu]

14 out of 225 runs failed -> 6% failure rate

[wenyangchu]

Hi @marcoabreu, can it be that the multithreading of this iterator does not guarantee the order of the patches?

[marcoabreu]

I would have to check, but that could be a good start for further investigation.

[larroy]

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11382/6/pipeline

[szha]

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11338/4/pipeline

[DickJC123]

I posted a reply to a different issue here, then removed it. Sorry for the confusion.

[perdasilva]

@marcoabreu this issue can now be closed =)

[lebeg]

@marcoabreu would you close this issue as it has been resolved?

[lebeg]

@marcoabreu ping