first commit

.gitignore

@@ -0,0 +1,24 @@
+venv/
+dist/
+build/
+*.eggs
+*.egg-info/
+*.pyc
+*/__pycache__/
+.pytest_cache/
+.mypy_cache/
+tags
+
+htmlcov/
+.coverage
+
+*.swp
+*.swo
+
+*.csv
+*.iml
+.idea
+*.code-workspace
+.vscode
+site/
+!src/epstats/toolkit/testing/resources/*.csv

.pre-commit-config.yaml

@@ -0,0 +1,19 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+ rev: v3.2.0
+ hooks:
+ # - id: trailing-whitespace
+ - id: end-of-file-fixer
+ - id: check-yaml
+ - id: check-added-large-files
+- repo: https://gitlab.com/pycqa/flake8
+ rev: 3.8.3
+ hooks:
+ - id: flake8
+- repo: https://github.com/odwyersoftware/brunette
+ rev: b8fc75f460885f986a01842664e0571769b2cc12
+ hooks:
+ - id: brunette
+ entry: brunette --config=setup.cfg

CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog #
+
+## Version 1.0.1 ##
+Features:
+- Add CHANGELOG
+- Final updates to README
+
+Bug fixes:

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Avast Software 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,39 @@
+artifactory="https://artifactory.srv.int.avast.com/artifactory/api/pypi/pypi/simple"
+
+brunette:
+ brunette src/epstats tests setup.py --check
+
+flake:
+ flake8 src/epstats tests setup.py
+
+test:
+ pytest
+
+check: brunette flake test
+
+install:
+ python -m pip install virtualenv
+ python -m virtualenv venv
+ source venv/bin/activate && python -m pip install -e .
+
+install-dev:
+ python -m pip install virtualenv
+ python -m virtualenv venv
+ source venv/bin/activate && python -m pip install -e ".[dev]" --index-url=${artifactory}
+ source venv/bin/activate && pre-commit install
+ source venv/bin/activate && python -m pip install ipykernel
+ source venv/bin/activate && ipython kernel install --user --name=ep-stats
+
+clean:
+ rm -rf build src/__pycache__ src/epstats/__pycache__ src/epstats/server/__pycache__ __pycache__ \
+ tests/__pycache__ tests/epstats/__pycache__ .pytest_cache src/*.egg-info .eggs tests/epstats/__pycache__\
+ tests/epstats/toolkit/__pycache__ tests/epstats/toolkit/testing/__pycache__ \
+ src/epstats/toolkit/__pycache__ src/epstats/toolkit/testing/__pycache__ \
+ src/epstats/toolkit/testing/resources/__pycache__ \
+ tests/epstats/server/__pycache__ tests/epstats/toolkit/__pycache__
+
+locust-evaluate:
+ locust -f src/epstats/locust.py --headless --host http:https://localhost:8080 -u 40 -r 5 --tags evaluate
+
+locust-health:
+ locust -f src/epstats/locust.py --headless --host https://localhost:8080 -u 1000 -r 50 --tags health

README.md

@@ -0,0 +1,129 @@
+<img src="theme/experiment_b.png" align="right" />
+
+# Ep-Stats
+
+**Statistical package for experimentation platform.**
+
+It provides a general python package and REST API that can be used to evaluate any metric
+in AB test experiment.
+
+## Features
+
+* Robust two-tailed t-test implementation with multiple p-value corrections and delta-methods applied.
+* Sequential evaluations allowing to stop experiments early.
+* Connect it to any data source to either get pre-aggregated or per randomization unit data.
+* Simple expression language to define arbitrary metrics.
+* REST API to integrate it as a service in experimentation portal with score cards.
+
+## Documentation
+
+We have got a lovely [documentation](https://git.int.avast.com/pages/doodlebug/ep-stats-lib/).
+
+## Base Example
+
+Ep-stats allows for quick experiment evaluation. We are using provided testing data to evaluate metric `Click-through Rate` in experiment `test-conversion`.
+
+```python
+from epstats.toolkit import Experiment, Metric, SrmCheck
+experiment = Experiment(
+ 'test-conversion',
+ 'a',
+ [Metric(
+ 1,
+ 'Click-through Rate',
+ 'count(test_unit_type.unit.click)',
+ 'count(test_unit_type.global.exposure)'),
+ ],
+ [SrmCheck(1, 'SRM', 'count(test_unit_type.global.exposure)')],
+ unit_type='test_unit_type')
+
+# This gets testing data, use other Dao or get aggregated goals in some other way.
+from epstats.toolkit.testing import TestData
+goals = TestData.load_goals_agg(experiment.id)
+
+# evaluate experiment
+ev = experiment.evaluate_agg(goals)
+```
+
+`ev` contains evaluations of exposures, metrics and checks. This will have following output.
+
+`ev.exposures`:
+
+| exp_id | exp_variant_id | exposures |
+| :----- | :------------- | --------: |
+|test-conversion|a|21|
+|test-conversion|b|26|
+
+`ev.metrics`:
+
+| exp_id | metric_id | metric_name | exp_variant_id | count | mean | std | sum_value | confidence_level | diff | test_stat | p_value | confidence_interval | standard_error | degrees_of_freedom |
+| :----- | --------: | :---------- | -------------: | ----: | ---: | --: | --------: | ---------------: | ---: | --------: | ------: | ------------------: | -------------: | -----------------: |
+|test-conversion|1|Click-through Rate|a|21|0.238095|0.436436|5|0.95|0|0|1|1.14329|0.565685|40|
+|test-conversion|1|Click-through Rate|b|26|0.269231|0.452344|7|0.95|0.130769|0.223152|0.82446|1.18137|0.586008|43.5401|
+
+`ev.checks`:
+
+| exp_id | check_id | check_name | variable_id | value |
+| :----- | -------: | :--------- | :---------- | ----: |
+|test-conversion|1|SRM|p_value|0.465803|
+|test-conversion|1|SRM|test_stat|0.531915|
+|test-conversion|1|SRM|confidence_level|0.999000|
+
+## Installation
+
+You can install this package via `pip`.
+
+```bash
+pip install ep-stats
+```
+
+## Running
+
+You can run a testing version of ep-stats via
+
+```bash
+python -m epstats
+```
+
+Then see Swagger on [http:https://localhost:8080/docs](http:https://localhost:8080/docs) for API documentation.
+
+## Contributing
+
+To get started locally, you can clone the repo and quickly get started using the `Makefile`.
+
+```bash
+git clone https://github.com/avast/ep-stats.git
+cd ep-stats
+make install-dev
+```
+
+It sets a new virtual environment `venv` in `./venv` using [venv](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/), installs all development dependencies, and sets [pre-commit](https://pre-commit.com/) git hooks to keep the code neatly formatted with [flake8](https://pypi.org/project/flake8/) and [brunette](https://pypi.org/project/brunette/).
+
+To run tests, you can use `Makefile` as well.
+
+```bash
+source venv/bin/activate # activate python environment
+make check
+```
+
+To run a development version of ep-stats do
+
+```bash
+source venv/bin/activate
+cd src
+python -m epstats
+```
+
+### Documentation
+
+To update documentation run
+
+```bash
+mkdocs gh-deploy
+```
+
+It updates documentation in GitHub pages stored in branch `gh-pages`.
+
+## Acknowledgement
+
+Software engineering practices of this package have been heavily inspired by marvelous [calmcode.io](https://calmcode.io/) site managed by [Vincent D. Warmerdam](https://github.com/koaning).

docs/api/check.md

@@ -0,0 +1,3 @@
+# Check
+
+::: epstats.toolkit.Check

docs/api/dao.md

@@ -0,0 +1,3 @@
+# Dao
+
+::: epstats.toolkit.Dao

docs/api/dao_factory.md

@@ -0,0 +1,3 @@
+# DaoFactory
+
+::: epstats.toolkit.DaoFactory

docs/api/evaluation.md

@@ -0,0 +1,3 @@
+# Evaluation
+
+::: epstats.toolkit.Evaluation

docs/api/experiment.md

@@ -0,0 +1,3 @@
+# Experiment
+
+::: epstats.toolkit.Experiment

docs/api/srm_check.md

@@ -0,0 +1,3 @@
+# SrmCheck
+
+::: epstats.toolkit.SrmCheck

docs/api/statistics.md

@@ -0,0 +1,3 @@
+# Statistics
+
+::: epstats.toolkit.statistics.Statistics

docs/api/test_data.md

@@ -0,0 +1,3 @@
+# Test Data
+
+::: epstats.toolkit.testing.TestData

docs/index.md

@@ -0,0 +1,57 @@
+<img src="images/experiment_b.png" align="right" />
+
+# Ep-Stats
+
+**Statistical package for experimentation platform.**
+
+It provides a general python package and REST API that can be used to evaluate any metric
+in AB test experiment.
+
+## Features
+
+* Robust two-tailed t-test implementation with multiple p-value corrections and delta-methods applied.
+* Sequential evaluations allowing to stop experiments early.
+* Connect it to any data source to either get pre-aggregated or per randomization unit data.
+* Simple expression language to define arbitrary metrics.
+* REST API to integrate it as a service in experimentation portal with score cards.
+
+We encourage all readers to get familiar with basic EP [Principles](principles.md) and then follow
+[Quick Start](user_guide/quick_start.md).
+
+## Architecture
+
+In regular experimentation platform, client data and telemetry have to pass through several components before experimenters can see results in metrics in scorecards. Ep-stats solves the statistical part of the pipeline as is described on following image.
+
+![](images/architecture.png)
+
+Client data and telemetry collection are specific to the company, we do not strive to provide any support for this part. *Aggregator* is optional part between raw data and ep-stats that can help to unify and pre-aggregate data consumed by ep-stats. *Scorecards* represent user interface in some kind of experimentation portal or knowledge base that lists experiments and displays scorecards with experiment results and statistics.
+
+Ep-stats offers following components:
+
+1. DAO (data access object) interfacing underlying data source with a way how to compile required metric data into SQL or anything else in use.
+1. Stats computing experiment evaluation with statistics.
+1. REST API a web app that makes it easy to integrate experiment evaluations in scorecards.
+
+## Known Limitations and Suggestion for Future Work
+
+Field of online experimentation is developing as well as data, metrics, methods, statistics. We strive to provide correct experiment evaluation. Its development takes time.
+
+Current main and known limitations.
+
+1. Metrics that are not based on (randomization) unit type (e.g. Views per User if session is our unit type) require application of
+delta method and bootstrapping[^1]. This is not implemented yet.
+1. Drill-down into dimensions is not implemented yet.
+1. Experiment data are aggregated as whole, cumulative evaluation or data for timeline graphs are not yet implemented.
+
+## Origin
+
+Ep-stats originated as a part of experimentation platform implementation in [Avast](https://www.avast.com).
+While there are many books on experimentation and statistics, there are few or none good implementations of it. We aim to fill in this gap between theory and practice by open-sourcing ep-stats package.
+
+We have been using EP with this implementation of ep-stats to run and evaluate hundreds of experiments in Avast. We will be adding new stuff here as we improve it and test it in Avast.
+
+## Inspiration
+
+Software engineering practices of this package have been heavily inspired by marvelous [calmcode.io](https://calmcode.io/) site managed by [Vincent D. Warmerdam](https://github.com/koaning).
+
+[^1]: [A. Deng et al., Applying the Delta Method in Metrics Analytics: A Practical Guide with Novel Ideas](https://arxiv.org/pdf/1803.06336.pdf)