Test failures and errors

[jfinkels]

Cloning this repository, installing the dependencies, then running nosetests yields three test errors and one test failure. The installation and testing output from the shell appears below. I am using Ubuntu 16.04 and Python 2.7.11.

---

$ git clone [email protected]:posterior/goftests
Cloning into 'goftests'...
remote: Counting objects: 89, done.        
remote: Total 89 (delta 0), reused 0 (delta 0), pack-reused 89        
Receiving objects: 100% (89/89), 55.89 KiB | 0 bytes/s, done.
Resolving deltas: 100% (46/46), done.
Checking connectivity... done.
$ cd goftests/
$ virtualenv .env -p python2
Running virtualenv with interpreter /usr/bin/python2
New python executable in /tmp/goftests/.env/bin/python2
Also creating executable in /tmp/goftests/.env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
/goftests$ . .env/bin/activate
(.env) /goftests$ pip install -r requirements.txt 
Collecting numpy (from -r requirements.txt (line 1))
  Using cached numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting scipy (from -r requirements.txt (line 2))
  Using cached scipy-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting scikit-learn (from -r requirements.txt (line 3))
  Using cached scikit_learn-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pep8 (from -r requirements.txt (line 4))
  Using cached pep8-1.7.0-py2.py3-none-any.whl
Collecting pyflakes (from -r requirements.txt (line 5))
  Using cached pyflakes-1.2.3-py2.py3-none-any.whl
Collecting nose (from -r requirements.txt (line 6))
  Using cached nose-1.3.7-py2-none-any.whl
Installing collected packages: numpy, scipy, scikit-learn, pep8, pyflakes, nose
Successfully installed nose-1.3.7 numpy-1.11.0 pep8-1.7.0 pyflakes-1.2.3 scikit-learn-0.17.1 scipy-0.17.1
(.env) $ pip freeze --local
nose==1.3.7
numpy==1.11.0
pep8==1.7.0
pkg-resources==0.0.0
pyflakes==1.2.3
scikit-learn==0.17.1
scipy==0.17.1
(.env) $ nosetests
...............................S......S.......S...............S.....................E..S....S......E................S.....SS...F.........E.S
======================================================================
ERROR: goftests.test.test_scipy_stats('invwishart',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/goftests/.env/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/goftests/goftests/test.py", line 166, in _test_scipy_stats
    params = [tuple(1.0 + rand(dist.numargs))]
AttributeError: 'invwishart_gen' object has no attribute 'numargs'

======================================================================
ERROR: goftests.test.test_scipy_stats('matrix_normal',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/goftests/.env/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/goftests/goftests/test.py", line 166, in _test_scipy_stats
    params = [tuple(1.0 + rand(dist.numargs))]
AttributeError: 'matrix_normal_gen' object has no attribute 'numargs'

======================================================================
ERROR: goftests.test.test_scipy_stats('wishart',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/goftests/.env/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/goftests/goftests/test.py", line 166, in _test_scipy_stats
    params = [tuple(1.0 + rand(dist.numargs))]
AttributeError: 'wishart_gen' object has no attribute 'numargs'

======================================================================
FAIL: goftests.test.test_scipy_stats('triang',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/goftests/.env/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/goftests/goftests/test.py", line 183, in _test_scipy_stats
    assert_greater(gof, TEST_FAILURE_RATE)
AssertionError: 0.0009710088328865929 not greater than 0.001
-------------------- >> begin captured stdout << ---------------------
param = (0.75580420415722394,)
    Prob    Count
   0.100      134 ------------------------------------------------------------
   0.100      128 ---------------------------------------------------------
   0.100      122 -------------------------------------------------------
   0.100      118 -----------------------------------------------------
   0.100      112 --------------------------------------------------
   0.100      108 ------------------------------------------------
   0.100      106 -----------------------------------------------
   0.100      104 -----------------------------------------------
   0.100       91 -----------------------------------------
   0.100       76 ----------------------------------
    Prob    Count
   0.100      136 ------------------------------------------------------------
   0.100      128 --------------------------------------------------------
   0.100      120 -----------------------------------------------------
   0.100      118 ----------------------------------------------------
   0.100      110 -------------------------------------------------
   0.100      110 -------------------------------------------------
   0.100      107 -----------------------------------------------
   0.100      104 ----------------------------------------------
   0.100       90 ----------------------------------------
   0.100       76 ----------------------------------
norm = 0.9969 +- 0.03015
     = 0.9969

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 140 tests in 65.038s

FAILED (SKIP=10, errors=3, failures=1)

[fritzo]

Thanks for reporting, @jfinkels This should be fixed by #2

[jfinkels]

That pull request seems to have fixed the 3 errors and 1 failure originally reported, but I now see two different failures:

$ nosetests
...............................S......S.......................S.....................S..S....S.F....S..F.............S.....SS.............S.S
======================================================================
FAIL: (goftests.test.test_scipy_stats('logistic',)) runTest
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jeffrey/src/gamalon/goftests/.env2/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/jeffrey/src/gamalon/goftests/goftests/test.py", line 204, in _test_scipy_stats
    assert_greater(gof, TEST_FAILURE_RATE)
AssertionError: 0.0003963727594426169 not greater than 0.0005
-------------------- >> begin captured stdout << ---------------------
param = ()
    Prob    Count
   0.100      141 ------------------------------------------------------------
   0.100      127 ------------------------------------------------------
   0.100      124 -----------------------------------------------------
   0.100      119 ---------------------------------------------------
   0.100      115 -------------------------------------------------
   0.100      101 -------------------------------------------
   0.100      100 -------------------------------------------
   0.100      100 -------------------------------------------
   0.100       88 -------------------------------------
   0.100       84 ------------------------------------

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: (goftests.test.test_scipy_stats('multivariate_normal',)) runTest
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jeffrey/src/gamalon/goftests/.env2/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/jeffrey/src/gamalon/goftests/goftests/test.py", line 204, in _test_scipy_stats
    assert_greater(gof, TEST_FAILURE_RATE)
AssertionError: 0.00045659701458544802 not greater than 0.0005
-------------------- >> begin captured stdout << ---------------------
param = (array([ 1.]), array([[ 1.]]))
    Prob    Count
   0.100      127 ------------------------------------------------------------
   0.100      119 --------------------------------------------------------
   0.100      114 ------------------------------------------------------
   0.100      112 -----------------------------------------------------
   0.100      110 ----------------------------------------------------
   0.100      106 --------------------------------------------------
   0.100      105 --------------------------------------------------
   0.100      103 -------------------------------------------------
   0.100      102 ------------------------------------------------
   0.100      101 ------------------------------------------------
    Prob    Count
   0.100      127 ------------------------------------------------------------
   0.100      119 --------------------------------------------------------
   0.100      114 ------------------------------------------------------
   0.100      110 ----------------------------------------------------
   0.100      109 ---------------------------------------------------
   0.100      109 ---------------------------------------------------
   0.100      105 --------------------------------------------------
   0.100      103 -------------------------------------------------
   0.100      102 ------------------------------------------------
   0.100      101 ------------------------------------------------
norm = 0.9995 +- 0.03015
     = 0.9995
param = (array([ 1.,  1.]), array([[ 1.,  0.],
       [ 0.,  1.]]))
    Prob    Count
   0.077      204 ------------------------------------------------------------
   0.077      196 ----------------------------------------------------------
   0.077      174 ---------------------------------------------------
   0.077      171 --------------------------------------------------
   0.077      166 -------------------------------------------------
   0.077      163 ------------------------------------------------
   0.077      161 -----------------------------------------------
   0.077      151 --------------------------------------------
   0.077      149 --------------------------------------------
   0.077      145 -------------------------------------------
   0.077      145 -------------------------------------------
   0.077      138 -----------------------------------------
   0.077      137 ----------------------------------------

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 140 tests in 66.564s

FAILED (SKIP=12, failures=2)

[danring]

With seed_all(0), it passes under scipy==0.15, but fails under scipy>=0.16. With seed_all(2) it seems to work with all, but that's not very satisfying.

Changing sample_count = 100 + 1000 * dim to sample_count = 200 + 1000 * dim seems to fix all.