Jon Sedar Personal Project
Experimental models to demonstrate pymc3
capabilities.
Where suitable, I will submit these Notebooks for inclusion in the pymc3 documentation
Note:
- Each Notebook is designed to be standalone with a suggested MVP condaenv install, but you can also just install the env for this project.
- This project is hosted publicly on GitHub.
- This README is MacOS oriented
This repo uses Python 3.6.*
Download Anaconda distro at https://www.continuum.io/downloads
Theano devs recommend using clang
(aka clang++
) by default which you can install using Xcode Command Line Tools
$> xcode-select --install
It's also possible to use g++ (gcc), but in my experience that's about 1/2 the speed.
e.g. install via Homebrew
$> brew update && brew upgrade `brew outdated`
$> brew install gcc
...optional cleanup of old versions afterwards
$> brew cleanup
Note:
- By default
gcc
is symlinked / points toclang
on MacOS - It is possible to install a particular version of
gcc
via Homebrew, but this gets installed to a versioned dir e.g./usr/local/Cellar/gcc/9.3.0_1/bin/gcc-9
and by default is not symlinked fromgcc
, rather it is available asgcc-9
- Regardless, it is best to use
clang
because theano is/was developed to this and is optimized (much faster) - Note you then don't need to configure the cxx in
.theanorc
, it usesclang
by default
[user]
name = $YOUR_USERNAME
email = $YOUR_EMAIL
[filter "lfs"]
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
process = git-lfs filter-process
required = true
[global]
device=cpu
$> jupyter notebook --generate-config
$> jupyter qtconsole --generate-config
$> jupyter nbconvert --generate-config
$> git clone https://github.com/jonsedar/pymc3_examples.git
$> cd pymc3_examples/
IMPORTANT NOTE: We're not using the latest python here because we want
stability for the pymc3
and theano
installs.
Pay extra attention to the .theanorc
config
$> conda env create --file condaenv_pymc3_examples.yml
$> activate pymc3_examples
Cheat sheet of conda commands available online here.
$> /.pip_install.sh
Requires NodeJS
$> jupyter labextension install @telamonian/theme-darcula
A quick and cheap way to confirm your binaries are good and the installation works!
View the BLAS / MKL install
$> python -c "import numpy as np; np.__config__.show()"
Example output...
mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
blas_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
blas_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
lapack_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
lapack_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/Users/jon/anaconda/envs/pymc3_examples/include']
$> python -c "import numpy as np; np.test()"
Example output...
7285 passed, 80 skipped, 167 deselected, 12 xfailed, 3 xpassed, 2 warnings in 355.51 seconds
$> python -c "import scipy as sp; sp.test()"
... many tests will run and output a results summary e.g. ...
Example output...
14522 passed, 1274 skipped, 1225 deselected, 79 xfailed, 10 xpassed, 25 warnings in 777.53 seconds
Method 1:
$> python -m pytest -xv --cov=pymc3 --cov-report=html pymc3/
Method 2:
Takes ~90mins on my Macbook 2017 :/
$> python -c "import pymc3 as pm; pm.test()"
Example output...
TBD
Takes forever, see installation docs
Quicker:
$> python -c "import theano; theano.test()"
Alternative (takes ~3 hours)
$> theano-nose -s
... many tests will run and output a results summary e.g. ...
As per https://stackoverflow.com/questions/40993626/list-memory-usage-in-ipython-and-jupyter
import sys
# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']
# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)
Remove unused packages and tarballs etc as explained https://conda.io/docs/commands/conda-clean.html
$> conda clean -v -a
NOTE @2019-01-19 there's an open bug in conda
on Windows that the ~/AppData/Local/Continuum/Anaconda3/pkgs/.trash
directory needs manual emptying afterwards
Ought to provide a speed-boost for MKL operations (many numpy, scipy operations under the hood). See docs
$> python -c "import mkl; mkl.set_num_threads(4); print(mkl.get_max_threads())
4
Interactive widgets are cool. Beware potential issue with lax requirements. Ensure conda_*.yaml
contains correct pairs of :
- ipywidgets==6.0.0
- widgetsnbextension==2.*
from ipywidgets import widgets
from IPython.display import display
text = widgets.Text()
text.on_submit(lambda x: print('hello {}'.format(x.value)))
display(text)
Straight breakpoint:
import pdb; pdb.set_trace()
Or print, often helpful if you can't set a breakpoint
for att in dir(myobject):
print(att, getattr(myobject, att))
See data/README_DATA.md
Jonathan Sedar © 2020