add benchmarks tutorials and simulation doc strings (#41)

* change permutations so it is one sided * add benchmarks tutorials and simulation doc strings * Update indep_sim.py
neurodata · Dec 11, 2019 · 4fc939d · 4fc939d
1 parent c6bcd1a
commit 4fc939d
Show file tree

Hide file tree

Showing 275 changed files with 909 additions and 305 deletions.
diff --git a/docs/README.md b/docs/README.md
@@ -0,0 +1,33 @@
+# Building Docs
+
+We currently use Sphinx
+
+If you only want to get the documentation, this can be found at [https://mgc.neurodata.io](https://mgc.neurodata.io).
+
+## Python Dependencies
+
+You will need to install all the dependencies as defined in `requirements.txt` file. The above can be installed by entering:
+
+ pip3 install -r requirements.txt
+
+in the `docs/` directory.
+
+## Pandoc dependency
+
+In addition, you need to install `pandoc` for `nbsphinx`. If you are on linux, you can enter:
+
+ sudo apt-get install pandoc
+
+If you are on macOS and have `homebrew` installed, you can enter:
+
+ brew install pandoc
+
+Otherwise, you can visit [pandoc installing page](https://pandoc.org/installing.html) for more information.
+
+## Generating the documentation
+
+To build the HTML documentation, enter:
+
+ make html
+
+in the `docs/` directory. If all goes well, this will generate a `build/html/` subdirectory containing the built documentation.
diff --git a/docs/_templates/footer.html b/docs/_templates/footer.html
@@ -0,0 +1,10 @@
+{% extends "!footer.html" %}
+{% block extrafooter %}
+<p style="text-align: center; margin: .5rem;">
+ <a href="https://www.netlify.com">
+ <img src="https://www.netlify.com/img/global/badges/netlify-color-accent.svg" />
+ </a>
+</p>
+{{ super() }}
+{% endblock %} 
+0 comments on commit 90d7854
diff --git a/docs/conf.py b/docs/conf.py
@@ -95,14 +95,14 @@
  "display_github": True,
  # Set the following variables to generate the resulting github URL for each page.
  # Format Template: https://{{ github_host|default("github.com") }}/{{ github_user }}/{{ github_repo }}/blob/{{ github_version }}{{ conf_py_path }}{{ pagename }}{{ suffix }}
- "github_user": "sampan501",
+ "github_user": "neurodata",
  "github_repo": "mgc",
  "github_version": "master/docs/",
 }
 
 linkcode_resolve = make_linkcode_resolve(
  "mgc",
- u"https://github.com/sampan501/"
+ u"https://github.com/neurodata/"
  "mgc/blob/{revision}/"
  "{package}/{path}#L{lineno}",
 )

diff --git a/docs/index.rst b/docs/index.rst
@@ -56,6 +56,7 @@ Documentation
 
  install
  reference/index
+ tutorials
  license
 
 .. toctree::

diff --git a/docs/reference/benchmarks.rst b/docs/reference/benchmarks.rst
diff --git a/docs/reference/sims.rst b/docs/reference/sims.rst
@@ -8,4 +8,20 @@ Independence Simulations
 .. autofunction:: linear
 .. autofunction:: exponential
 .. autofunction:: cubic
+.. autofunction:: joint_normal
+.. autofunction:: step
+.. autofunction:: quadratic
+.. autofunction:: w_shaped
 .. autofunction:: spiral
+.. autofunction:: uncorrelated_bernoulli
+.. autofunction:: logarithmic
+.. autofunction:: fourth_root
+.. autofunction:: sin_four_pi
+.. autofunction:: sin_sixteen_pi
+.. autofunction:: square
+.. autofunction:: two_parabolas
+.. autofunction:: circle
+.. autofunction:: ellipse
+.. autofunction:: diamond
+.. autofunction:: multiplicative_noise
+.. autofunction:: multimodal_independence
diff --git a/docs/tutorials.rst b/docs/tutorials.rst
@@ -0,0 +1,14 @@
+Tutorials
+*********
+
+Benchmarks
+==========
+These tutorials overview how to use the simulations and outlines comparisons
+between implemented tests in statistical power and computational performance.
+
+.. toctree::
+ :maxdepth: 1
+
+ tutorials/benchmarks/indep_power_sampsize
+ tutorials/benchmarks/indep_power_dimension
+ tutorials/benchmarks/simulations
diff --git a/benchmarks/__init__.py → docs/tutorials/benchmarks/__init__.py b/benchmarks/__init__.py → docs/tutorials/benchmarks/__init__.py
diff --git a/benchmarks/indep_power_dimension.ipynb → ...ls/benchmarks/indep_power_dimension.ipynb b/benchmarks/indep_power_dimension.ipynb → ...ls/benchmarks/indep_power_dimension.ipynb
@@ -1,5 +1,19 @@
 {
  "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Independence Tests Power over Increasing Dimension"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "These are same useful functions to import. Since we are calculating the statistical power over all the tests for all the simulations, we can just use a wild card import from the respective modules"
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 1,
@@ -31,6 +45,13 @@
  "sns.set(color_codes=True, style='white', context='talk', font_scale=2, palette=\"colorblind\")"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "These are some constants that are used in this notebook. If running these notebook, please only manipulate these constants if you are not running more tests. They define the number of replications. The simulations tested over and the independence tests tested over are defined also."
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 3,
@@ -78,6 +99,13 @@
  "]"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "These helper functions return the number of dimensions to test over and the range for each simulation. Limits were chosen based on when it is clear to determine that the tests were tending to 0 statistical power."
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 5,
@@ -113,6 +141,15 @@
  " return dim_range"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The following function calculates the estimated power ``POWER_REPS`` number off times and averages them. It does this iterating over the number of sample sizes.\n",
+ "\n",
+ "**Note: We only recommend running this code if running the next 2 cells ONCE to generate the csv files used to visualize the plots. This code takes a very long time to run and if running, we recommend using a machine with many cores.**"
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 6,
@@ -132,77 +169,22 @@
  },
  {
  "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "scrolled": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 48 concurrent workers.\n",
- "[Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 1.4min\n",
- "[Parallel(n_jobs=-1)]: Done 2 tasks | elapsed: 1.8min\n",
- "[Parallel(n_jobs=-1)]: Done 3 tasks | elapsed: 1.8min\n",
- "[Parallel(n_jobs=-1)]: Done 4 tasks | elapsed: 1.8min\n",
- "[Parallel(n_jobs=-1)]: Done 5 tasks | elapsed: 1.9min\n",
- "[Parallel(n_jobs=-1)]: Done 7 out of 100 | elapsed: 2.0min remaining: 26.8min\n",
- "[Parallel(n_jobs=-1)]: Done 9 out of 100 | elapsed: 2.2min remaining: 22.0min\n",
- "[Parallel(n_jobs=-1)]: Done 11 out of 100 | elapsed: 2.4min remaining: 19.2min\n",
- "[Parallel(n_jobs=-1)]: Done 13 out of 100 | elapsed: 3.8min remaining: 25.1min\n",
- "[Parallel(n_jobs=-1)]: Done 15 out of 100 | elapsed: 3.8min remaining: 21.7min\n",
- "[Parallel(n_jobs=-1)]: Done 17 out of 100 | elapsed: 4.0min remaining: 19.7min\n",
- "[Parallel(n_jobs=-1)]: Done 19 out of 100 | elapsed: 4.3min remaining: 18.5min\n",
- "[Parallel(n_jobs=-1)]: Done 21 out of 100 | elapsed: 4.8min remaining: 18.2min\n",
- "[Parallel(n_jobs=-1)]: Done 23 out of 100 | elapsed: 5.3min remaining: 17.9min\n",
- "[Parallel(n_jobs=-1)]: Done 25 out of 100 | elapsed: 5.5min remaining: 16.6min\n",
- "[Parallel(n_jobs=-1)]: Done 27 out of 100 | elapsed: 5.9min remaining: 16.0min\n",
- "[Parallel(n_jobs=-1)]: Done 29 out of 100 | elapsed: 7.4min remaining: 18.2min\n",
- "[Parallel(n_jobs=-1)]: Done 31 out of 100 | elapsed: 8.0min remaining: 17.8min\n",
- "[Parallel(n_jobs=-1)]: Done 33 out of 100 | elapsed: 8.5min remaining: 17.3min\n",
- "[Parallel(n_jobs=-1)]: Done 35 out of 100 | elapsed: 9.4min remaining: 17.4min\n",
- "[Parallel(n_jobs=-1)]: Done 37 out of 100 | elapsed: 9.6min remaining: 16.4min\n",
- "[Parallel(n_jobs=-1)]: Done 39 out of 100 | elapsed: 9.9min remaining: 15.4min\n",
- "[Parallel(n_jobs=-1)]: Done 41 out of 100 | elapsed: 10.4min remaining: 15.0min\n",
- "[Parallel(n_jobs=-1)]: Done 43 out of 100 | elapsed: 11.7min remaining: 15.6min\n",
- "[Parallel(n_jobs=-1)]: Done 45 out of 100 | elapsed: 11.9min remaining: 14.5min\n",
- "[Parallel(n_jobs=-1)]: Done 47 out of 100 | elapsed: 12.7min remaining: 14.3min\n",
- "[Parallel(n_jobs=-1)]: Done 49 out of 100 | elapsed: 13.7min remaining: 14.3min\n",
- "[Parallel(n_jobs=-1)]: Done 51 out of 100 | elapsed: 14.1min remaining: 13.6min\n",
- "[Parallel(n_jobs=-1)]: Done 53 out of 100 | elapsed: 15.2min remaining: 13.5min\n",
- "[Parallel(n_jobs=-1)]: Done 55 out of 100 | elapsed: 16.0min remaining: 13.1min\n",
- "[Parallel(n_jobs=-1)]: Done 57 out of 100 | elapsed: 17.5min remaining: 13.2min\n",
- "[Parallel(n_jobs=-1)]: Done 59 out of 100 | elapsed: 18.3min remaining: 12.7min\n",
- "[Parallel(n_jobs=-1)]: Done 61 out of 100 | elapsed: 18.7min remaining: 11.9min\n",
- "[Parallel(n_jobs=-1)]: Done 63 out of 100 | elapsed: 19.1min remaining: 11.2min\n",
- "[Parallel(n_jobs=-1)]: Done 65 out of 100 | elapsed: 21.9min remaining: 11.8min\n",
- "[Parallel(n_jobs=-1)]: Done 67 out of 100 | elapsed: 23.2min remaining: 11.4min\n",
- "[Parallel(n_jobs=-1)]: Done 69 out of 100 | elapsed: 24.8min remaining: 11.1min\n",
- "[Parallel(n_jobs=-1)]: Done 71 out of 100 | elapsed: 27.0min remaining: 11.0min\n",
- "[Parallel(n_jobs=-1)]: Done 73 out of 100 | elapsed: 31.2min remaining: 11.5min\n",
- "[Parallel(n_jobs=-1)]: Done 75 out of 100 | elapsed: 32.2min remaining: 10.7min\n",
- "[Parallel(n_jobs=-1)]: Done 77 out of 100 | elapsed: 33.3min remaining: 9.9min\n",
- "[Parallel(n_jobs=-1)]: Done 79 out of 100 | elapsed: 34.5min remaining: 9.2min\n",
- "[Parallel(n_jobs=-1)]: Done 81 out of 100 | elapsed: 37.0min remaining: 8.7min\n",
- "[Parallel(n_jobs=-1)]: Done 83 out of 100 | elapsed: 49.2min remaining: 10.1min\n",
- "[Parallel(n_jobs=-1)]: Done 85 out of 100 | elapsed: 50.9min remaining: 9.0min\n",
- "[Parallel(n_jobs=-1)]: Done 87 out of 100 | elapsed: 52.3min remaining: 7.8min\n",
- "[Parallel(n_jobs=-1)]: Done 89 out of 100 | elapsed: 53.8min remaining: 6.7min\n",
- "[Parallel(n_jobs=-1)]: Done 91 out of 100 | elapsed: 55.6min remaining: 5.5min\n",
- "[Parallel(n_jobs=-1)]: Done 93 out of 100 | elapsed: 58.1min remaining: 4.4min\n",
- "[Parallel(n_jobs=-1)]: Done 95 out of 100 | elapsed: 63.4min remaining: 3.3min\n",
- "[Parallel(n_jobs=-1)]: Done 97 out of 100 | elapsed: 64.7min remaining: 2.0min\n",
- "[Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed: 83.7min finished\n"
- ]
- }
- ],
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
  "source": [
  "outputs = Parallel(n_jobs=-1, verbose=100)(\n",
  " [delayed(estimate_power)(sim, test) for sim in simulations for test in tests]\n",
  ")"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The following code loops over each saved independence test file and generates absolute power curves for each test and for each simulation modality."
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 8,
@@ -292,13 +274,6 @@
  "source": [
  "plot_power()"
  ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
  }
  ],
  "metadata": {

diff --git a/benchmarks/indep_power_sampsize.ipynb → ...als/benchmarks/indep_power_sampsize.ipynb b/benchmarks/indep_power_sampsize.ipynb → ...als/benchmarks/indep_power_sampsize.ipynb
@@ -4,7 +4,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "# Power over Increasing Sample Size"
+ "# Independence Tests Power over Increasing Sample Size"
  ]
  },
  {
@@ -162,7 +162,7 @@
  },
  {
  "cell_type": "code",
- "execution_count": 9,
+ "execution_count": 1,
  "metadata": {},
  "outputs": [],
  "source": [
@@ -250,13 +250,6 @@
  "source": [
  "plot_power()"
  ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
  }
  ],
  "metadata": {

diff --git a/benchmarks/power.py → docs/tutorials/benchmarks/power.py b/benchmarks/power.py → docs/tutorials/benchmarks/power.py