Create a directory called 'data' in the home directory of the repo but running: mkdir data

Go to https://www.rxrx.ai/rxrx19a, scroll to the bottom of the page, and download the "Metadata" and "Deep Learning Embeddings." Put these files (metadata.csv and embeddings.csv) into the data directory.

Create a Python3 virtual environment and install the packages from the provided requirements.txt file.

This can be done by running the following commands:
python3 -m venv env
source env/bin/activate
python3 -m pip install -r requirements.txt

If that does not work, just make sure these packages are installed:

pandas
numpy
imbalanced-learn
scikit-learn
matplotlib
shap

These scripts run using the SLURM Workload Manager. To perform these analyses without SLURM, you must replace sbatch with bash in all the bash scripts. When running with SLURM, you may need to adjust the bash scripts resource requests or other arguments.

Run this command to split the data by experiment and calculate SHAP values for each feature when predicting plate (batch effect/noise) vs disease condition (biological signal):
bash bash_scripts/split_n_get_shaps.sh data/embeddings.csv data/metadata.csv

If you want to normalize the deep learning embeddings to remove batch effects (we strongly recommend this) then add a final argument n when running, like so:
bash bash_scripts/split_n_get_shaps.sh data/embeddings.csv data/metadata.csv n

If you want to run with just feature dropping, instead add the argument f. If you want to use both, add b instead.

If feature dropping, run this command to view plots of the two SHAP values for each feature plotted against each other:
bash bash_scripts/run_shap_plots.sh

Look at each experiments' plot to determine a cutoff for the disease condition shap that will remove the features that have small SHAP values for disease condition but have large shap values for plate. Record these four cut off values because we will use them in subsequent scripts.

Run this command to perform a sensitivity test on each experiment except replace the four numbers with your determined SHAP cut off values for the HRCE-1, HRCE-2, VERO-1, and VERO-2 experiments, respectively. You can skip this step if you are only normalizing and not feature dropping.
bash bash_scripts/run_all_sensitivity_tests.sh 0.0002 0.0022 0.002 0.00015

Run this command to identify treatments that might be effective against SARS-CoV-2, using two different methods:
bash bash_scripts/identify_effective_treatments.sh 0.0002 0.0022 0.002 0.00015

If you want to identify effective compounds without feature dropping, run the above script with all cutoffs as 0 like so:
bash bash_scripts/identify_effective_treatments.sh n

SHAP Value Cutoff: 0.0002

SHAP Value Cutoff: 0.001

SHAP Value Cutoff: 0.001

SHAP Value Cutoff: 0.00125