wMMD is a Python module for implementing word-level maximum mean discrepancy (wMMD) regularization.

wMMD requires Python 3.9 + Python libraries:

pip install -r requirements.txt

You can check the latest code with the command:

git clone https://github.com/youqiangao/wMMD.git

To train a CNN model on the ChileEarthquakeT1 dataset with the wMMD (weight = 10^3), l1 (weight = 10^3) or dropout (dropout rate = 0.5), repsectively execute the following commands:

python main.py --dataset chile-earthquakeT1 --model cnn --regularization wmmd --weight-exponent 3

python main.py --dataset chile-earthquakeT1 --model cnn --regularization l1 --weight-exponent 3

python main.py --dataset chile-earthquakeT1 --model cnn --structure dropout --dropout-rate 0.5

To check more arguments of the commands, run

python main.py --help

The regularizer.py file contains the implementation of the wMMD regularization. To train your models using wMMD regularization, simply include an additional line in your Python code.

loss = loss - weight * wMMD(model.embedding.weight, stopping_idx)

The weight hyperparameter balances the objective loss function and regularization loss. The stopping_idx includes the indices you wish to exclude from the calculation of wMMD regularization. By default, it is set to [0], representing the index for padding.

• Three different neural network architectures are considered: GRU, BiLSTM, and CNN.
• The effects of various embedding sizes are investigated with $r$ = 20, 50, 100, 200, 300, as well as one pre-trained embedding with a fixed size of 300.
• For L1 and wMMD, the optimal weight $\lambda$ is determined by a grid search over $\lbrace 10^{(s-16)/5}, s = 1, \dots, 36 \rbrace$. As for dropout, the best dropout rate is tuned by a grid search over $\lbrace 0.05s, s = 1, \dots, 19 \rbrace$. The prediction performance of all methods is measured by classification accuracy.

• We report average classification accuracy and standard error (in parenthesis) of 20 replications on ChileEarthquakeT1. The highest performance in each setting is highlighted in bold.
• The pretrained embedding denoted as SBWC can be found in this link.

• Average classification accuracy and standard error (in parenthesis) is presented in the following table, based on 10 replications on the BBC News dataset. The highest performance in each setting is highlighted in bold.
• The pretrained embedding denoted as GoogleNews can be found in this link.

• The following figure displays average classification accuracy and 95% confidence interval of CNN trained on ChileEarthquakeT1 using various hyperparameter values. The embedding size is 20. The gray dashed line represents the average accuracy with no regularization.
• In constrst to dropout and L1, the performance with wMMD regularization initially increases and then stabilizes as the weight increases. Furthermore, wMMD exhibits a broader range of optimal weights compared to L1. These properties significantly mitigate the hyperparameter-tuning issue. It is noteworthy that similar trends are observed for other models and embedding sizes on both datasets in the conducted experiments.