This project is an experimental repository focusing on dealing with datasets containing a high level of noisy labels (50% and above). This repository features experiments conducted on the FashionMNIST and CIFAR datasets using the ResNet34 as the baseline classifier.

The repository explores various training strategies (trainers), including ForwardLossCorrection, CoTeaching, JoCoR, and O2UNet. Specifically, for datasets with unknown transition matrices, DualT is employed as the Transition Matrix Estimator. Given the computational complexity and practical performance considerations, our experiments primarily focus on ForwardLossCorrection and CoTeaching. We conducted multiple experiments with different random seeds to compare these two methods.

Initial explorations on FashionMNIST0.5 with JoCoR and O2UNet have shown promising results. This repository serves as a resource for those interested in robust machine learning techniques under challenging conditions of high label noise.

A brief pipeline:

• Datasets (3 classes: 0, 1, 2; instead of 10):

  • FashionMNIST with Known Flip Rate

    • Noise Level: 0.5

    Transition Matrix of FashionMNIST0.5

    0.5	0.2	0.3
    0.3	0.5	0.2
    0.2	0.3	0.5

    
    Noisy Data

    
    Clean Data

    • Noise Level: 0.6

    Transition Matrix of FashionMNIST0.6

    0.4	0.3	0.3
    0.3	0.4	0.3
    0.3	0.3	0.4

    
    Samples in FashionMNIST0.6

  • CIFAR with Unknown Flip Rate

    
    Samples in CIFAR

• Base Classifier:

  • ResNet-34

• Basic Robust Method(s):

  • Data Augmentation

• Robust Trainers:

  • Loss correction: ForwardLossCorrection
    • Includes: SymmetricCrossEntropyLoss
  • Multi-network learning: CoTeaching
  • Multi-network learning: JoCoR
  • Multi-round learning: O2UNet

• Transition Matrix Estimator:

  • Dual-T

According to the the loss trends, we find that our robust trainers may also act as regularizers to avoid overfitting.

• ForwardLossCorrection

Loss Trend

• CoTeaching

Loss Trend

We have conducted a series of experiments utilizing 10 distinct random seeds to evaluate the performance of ForwardLossCorrection and CoTeaching. Below is a detailed comparison of their performances.

It becomes evident from our analysis that CoTeaching exhibits superior performance under conditions of low noise. However, as the noise level escalates, ForwardLossCorrection demonstrates enhanced robustness, outperforming CoTeaching.

In our preliminary experiments, both JoCoR and O2UNet showed promising results on the FashionMNIST0.5 dataset. Nevertheless, due to the substantial computational demands and the marginal improvements they offered over CoTeaching, we decided not to proceed with extensive experimentation on these methods.

• Estimation on FashionMNIST05

• Estimation on FashoinMNIST06

• Estimation on CIFAR