This is the code of paper Federated Learning on Non-IID Data Silos: An Experimental Study.

This code runs a benchmark for federated learning algorithms under non-IID data distribution scenarios. Specifically, we implement 4 federated learning algorithms (FedAvg, FedProx, SCAFFOLD & FedNova), 3 types of non-IID settings (label distribution skew, feature distribution skew & quantity skew) and 9 datasets (MNIST, Cifar-10, Fashion-MNIST, SVHN, Generated 3D dataset, FEMNIST, adult, rcv1, covtype).

Our follow-up works based on NIID-Bench:

• FedOV: Towards Addressing Label Skews in One-Shot Federated Learning (ICLR 2023)

• FedConcat: Exploiting Label Skew in Federated Learning with Model Concatenation (AAAI 2024)

We publish NIID-Bench challenge https://niidbench.xtra.science, a benchmark to compare federated learning algorithms on comprehensive non-IID data settings. Researchers are welcome to test their algorithms on these settings, upload their codes and participate in our leaderboard!

Implement partition.py to divide tabular datasets (csv format) into multiple files using our non-IID partitioning strategies. Column Class in the header is recognized as label. See an running example in partition_to_file.sh. The example dataset is Credit Card Fraud Detection.

To adapt to your own tabular dataset in ​partition.py, you need the following steps:

1. Load your own dataset in arrays. Replace Line 117-126.
2. The whole tabular dataset is stored in ​dataset​​. The label column ID is stored in class_id​​. Change Line 130 to your own label identifier.

If your dataset is image dataset, partition.py​​ is no longer applicable. You can refer to our function partition_data​​ in utils.py. You need to design your own dataloader like Line 183-198. For example, in load_mnist_data (Line 40), you need to write a dataloader to return your dataset as tuple (X_train, y_train, X_test, y_test). In terms of the dataloader format, you can refer to class MNIST_truncated​​ (Line 60 in dataset.py). After you get (X_train, y_train, X_test, y_test), the partition_data function will return the ​net_dataidx_map​.

To support more settings and faciliate future researches, we now integrate MOON. We add CIFAR-100 and Tiny-ImageNet.

You can download Tiny-ImageNet here. Then, you can follow the instructions to reformat the validation folder.

• Quantity-based label imbalance: each party owns data samples of a fixed number of labels.
• Distribution-based label imbalance: each party is allocated a proportion of the samples of each label according to Dirichlet distribution.

• Noise-based feature imbalance: We first divide the whole dataset into multiple parties randomly and equally. For each party, we add different levels of Gaussian noises.
• Synthetic feature imbalance: For generated 3D data set, we allocate two parts which are symmetric of(0,0,0) to a subset for each party.
• Real-world feature imbalance: For FEMNIST, we divide and assign the writers (and their characters) into each party randomly and equally.

• While the data distribution may still be consistent amongthe parties, the size of local dataset varies according to Dirichlet distribution.

Here is one example to run this code:

python experiments.py --model=simple-cnn \
    --dataset=cifar10 \
    --alg=fedprox \
    --lr=0.01 \
    --batch-size=64 \
    --epochs=10 \
    --n_parties=10 \
    --mu=0.01 \
    --rho=0.9 \
    --comm_round=50 \
    --partition=noniid-labeldir \
    --beta=0.5\
    --device='cuda:0'\
    --datadir='./data/' \
    --logdir='./logs/' \
    --noise=0 \
    --sample=1 \
    --init_seed=0

You can call function get_partition_dict() in experiments.py to access net_dataidx_map. net_dataidx_map is a dictionary. Its keys are party ID, and the value of each key is a list containing index of data assigned to this party. For our experiments, we usually set init_seed=0. When we repeat experiments of some setting, we change init_seed to 1 or 2. The default value of noise is 0 unless stated. We list the way to get our data partition as follow.

• Quantity-based label imbalance: partition=noniid-#label1, noniid-#label2 or noniid-#label3
• Distribution-based label imbalance: partition=noniid-labeldir, beta=0.5 or 0.1
• Noise-based feature imbalance: partition=homo, noise=0.1 (actually noise does not affect net_dataidx_map)
• Synthetic feature imbalance & Real-world feature imbalance: partition=real
• Quantity Skew: partition=iid-diff-quantity, beta=0.5 or 0.1
• IID Setting: partition=homo
• Mixed skew: partition = mixed for mixture of distribution-based label imbalance and quantity skew; partition = noniid-labeldir and noise = 0.1 for mixture of distribution-based label imbalance and noise-based feature imbalance.

Here is explanation of parameter for function get_partition_dict().

Note that the accuracy shows the average of three experiments, while the training curve is based on only one experiment. Thus, there may be some difference. We show the training curve to compare convergence rate of different algorithms.

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

• Cifar-10, 100 parties, sample rate = 0.1, batch size = 64, learning rate = 0.01

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

• Cifar-10, 100 parties, sample rate = 0.1, batch size = 64, learning rate = 0.01

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

• Cifar-10, 10 parties, sample rate = 1, batch size = 64, learning rate = 0.01

• Cifar-10, 100 parties, sample rate = 0.1, batch size = 64, learning rate = 0.01

If you find this repository useful, please cite our paper:

@inproceedings{li2022federated,
      title={Federated Learning on Non-IID Data Silos: An Experimental Study},
      author={Li, Qinbin and Diao, Yiqun and Chen, Quan and He, Bingsheng},
      booktitle={IEEE International Conference on Data Engineering},
      year={2022}
}