This is a benchmarking repository accompanying the obnb Python package.

conda create -n obnb python=3.8 -y && conda activate obnb

# Install PyTorch and PyG with CUDA 11.7
conda install pytorch=2.0.1 torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
conda install pyg=2.3.0 -c pyg -y

pip install obnb[ext]==0.1.0  # install obnb with extension modules (PecanPy, GraPE, ...)
pip install -r requirements_extra.txt  # extra dependencies for benchmarking

conda clean --all -y  # clean up

The extra dependencies are, e.g.,

• Hydra for managing experiments.
• Lightning for organizing model training framework.
• WandB for logging metrics.

Note: if you do not need to run the benchmarking experiments and only want to play around with our benchmarking results with one of the notebooks, you can skip the installation for PyTorch and PyG.

pip install obnb[ext]==0.1.0

Run get_data.py to download and set up data for all the experiments. Data will be saved under the datasets/ directory by default, and will take up approximately 6 GB of space.

python get_data.py

This step is completely optional and directly runing the training script will work fine. But runing get_data.py once before training prevents multiple parallel jobs doing the same data preprocessing work if the processed data is not available yet.

After setting up the data, one can run a single experiment by specifying the choices of network, label, and model:

python main.py dataset.network=BioGRID dataset.label=DisGeNET model=GCN

Check out the conf/model/ directory for all available model presets. The main model presets are:

• GCN
• GAT
• GCN+BoT
• GAT+BoT
• LogReg+Adj
• LogReg+Node2vec
• LogReg+Walklets

cd run

# GNN node feature ablation (example of runing GCN with node2vec features on BioGRID)
sh run_abl_gnn_feature.sh GCN BioGRID Node2vec

# C&S ablation (example of runing GCN with C&S post processing on BioGRID)
sh run_abl_cs.sh GCN BioGRID

# GNN label reuse ablation (example of runing GCN with label reuse on BioGRID)
sh run_abl_gnn_label.sh GCN BioGRID

# GNN label reuse with C&S ablation (example of runing GCN with label reuse with C&S on BioGRID)
sh run_abl_gnn_cs_label.sh GCN BioGRID

# GNN with bag of tricks, i.e., node2vec node feature + label reuse + C&S
sh run_gnn_bot.sh GCN BioGRID

To run all experiments presented in the paper (may take several days):

sh run_all.sh

First create a sweep agent, e.g., for BioGRID-DisGeNET-GCN:

wandb sweep conf/tune/BioGRID-DisGeNET-GCN.yaml

Then, follow the instruction from the command above to spawn sweep agents to automatically tune the model configuration on a particular dataset.

To run the notebooks, first download our benchmarking results (or you can rerun all the benchmarking experiments yourself using our run scripts described above).

wget -O results/main.csv.gz https://zenodo.org/record/8048305/files/main.csv.gz