This Repo contains the official implementation of the following paper:
| Venue | Method | Paper Title |
|---|---|---|
| ICLR'23 | FedDecorr | Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning |
and unofficial implementation of the following papers:
| Venue | Method | Paper Title |
|---|---|---|
| AISTATS'17 | FedAvg | Communication-Efficient Learning of Deep Networks from Decentralized Data |
| ArXiv'19 | FedAvgM | Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification |
| MLSys'20 | FedProx | Federated Optimization in Heterogeneous Networks |
| NeurIPS'20 | FedNova | Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization |
| CVPR'21 | MOON | Model-Contrastive Federated Learning |
| ICLR'21 | FedAdagrad/Yogi/Adam | Adaptive Federated Optimization |
| KDD'21 | FedRS | FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data |
| ICML'22 | FedLogitCal | Federated Learning with Label Distribution Skew via Logits Calibration |
| ICML'22/ECCV'22 | FedSAM | Generalized Federated Learning via Sharpness Aware Minimization/Improving Generalization in Federated Learning by Seeking Flat Minima |
| ICLR'23 | FedExp | FedExP: Speeding up Federated Averaging via Extrapolation |
TinyImageNet:
- Download the dataset to "data" directory from this link: https://cs231n.stanford.edu/tiny-imagenet-200.zip
- Unzip the downloaded file under "data" directory.
- Lastly, to reformat the validation set, under the folder "data/tiny-imagenet-200", run:
python3 preprocess_tiny_imagenet.py
Shell scripts to reproduce experimental results in our paper are under "run_scripts" folder. Simply changing the "ALPHA" variable to run under different degree of heterogeneity.
Here are commands that replicate our results:
FedAvg on CIFAR10:
bash run_scripts/cifar10_fedavg.sh
FedAvg + FedDecorr on CIFAR10:
bash run_scripts/cifar10_fedavg_feddecorr.sh
Experiments on other methods (FedAvgM, FedProx, MOON) and other datasets (CIFAR100, TinyImageNet) follow the similar manner.
If you find our repo/paper helpful, please consider citing our work :)
@article{shi2022towards,
title={Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning},
author={Shi, Yujun and Liang, Jian and Zhang, Wenqing and Tan, Vincent YF and Bai, Song},
journal={arXiv preprint arXiv:2210.00226},
year={2022}
}
Yujun Shi ([email protected])
Some of our code is borrowed following projects: MOON, NIID-Bench, SAM(Pytorch)