This python project is a helper package that uses power analysis to calculate required sample size for any experiment.

Sample size script lets you get the sample size estimation easily by providing metric inputs.

Please make sure you have Python 3 installed before using the script.

Verify Python was installed

python -V # python version should >=3.7.1, <3.11

Verify pip was installed

pip -V 

pip install sample-size
pip show sample-size # verify package was installed

run-sample-size will prompt required questions for you to enter the input it needs

run-sample-size

graph TD
    A(Alpha) --> B(Variants)
    B --> C(Metric Type)
    C --> D(Metadata)
    D --> E(MDE)
    E --> G(Alternative)
    G --> F{{Register another metric?}}
    F --> C & H(Sample Size)

• This package supports

  • Single and multiple metrics per calculation
  • Multiple cohorts, i.e. more than one treatment variant, per calculation
  • Metric types: Boolean, Numeric, and Ratio

• Default statistical power (80%) is used in run-sample-size all the time

• Input constraints

  • alpha: (0, 0.4]
  • probability (Boolean Metric): (0, 1)
  • variance (Numeric and Ratio Metrics): [0, )
  • registered metrics: [1, ]
  • variants: [2, ]

  Please be aware that we are running simulations many times when calculating sample size for multiple metrics or variants. Therefore, too many cohorts or metrics will have extremely long runtime.

All contributors and contributions are welcome! Please see the contributing docs for more information.