This repository implements experiments/baselines for SuperLim 2.

1. wget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh
2. bash Anaconda3-2022.10-Linux-x86_64.sh
3. yes, yes, yes
4. conda create -n ptgpu_venv python=3.9
5. conda activate ptgpu_venv
6. conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

1. git clone [email protected]:JoeyOhman/SuperLim-2-Testing.git
2. cd SuperLim-2-Testing
3. pip install -r requirements.txt
4. tmux new -s exp
5. tmux attach -t exp

1. Download all data to data directory 
2. Setup wandb key in api_wandb_key.txt
3. Configure GPUs to use in run_bert_experiments.sh
4. Specify models and tasks in run_bert_experiments.sh
5. Specify accumulation sizes for models in bert/bert_experiment_driver.py to suit your available GPU memory.
6. ./run_dummy_experiments.sh
7. ./run_bert_experiments.sh (this includes gpt experiments if wanted)
8. Run the collect_results/create_table.py script, it will collect results in results/experiments/metrics directory and create the results/experiments/metrics/model_deliverables directory with json files. 

This will fine-tune 672 models in total, if the 4 gpt-models are included (they are not by default). 84 models will be saved, one for each task-model combination.

1. Individual experiment results: results/experiments/metrics/
2. Packed results grouped by model: results/experiments/metrics/model_deliverables/
3. Model predictions on dev and test files: results/experiments/predictions/
4. Best fine-tuned models: results/experiments/models/

The tasks are named slightly different in this repository since it was developed during the development of SuperLim 2. See task_to_info_dict in Experiment.py for the task names used. When the final results files are created, these names are mapped to the official names.

Note: Winogender will be evaluated automatically when training/evaluating with SweMNLI.

The class hierarchy handles much of the execution flow, like a framework.

• paths.py defines paths that can be imported, and should work cross-platform and in docker containers etc
• compute_metrics.py implements functions to compute metrics required for the experiments
• collect_metrics.py loads the experiment results in the pre-defined results file hierarchy and plots a table.

Experiment.py defines the abstract class Experiment, which handles the experiment execution flow, keeps track of the model name, task name, and other meta-data. It also handles the writing of results to file, and contains task_to_info_dict which should contain the required meta-data for each task, such as which metric is used.

ExperimentBert.py defines another abstract class ExperimentBert, which handles loading of tokenizers and models, default hyperparameters, hyperparameter tuning, HuggingFace Trainer, Evaluation, etc.

Children of ExperimentBert simply define the task name, which should be supported in Experiment.py, and implements the abstract method create_dataset, which calls the _load_data method and preprocesses it to make it ready for training.

Inherits Experiment, and uses sklearn.dummy to provide a dummy baseline. This works for any task that has training data and is a classification or regression problem.

1. Make sure the data is accessible, preferably via HF Datasets, or otherwise through JSONL/TSV files
2. Create a function reformat_dataset_<task_name> in dataset_loader.py, that converts the dataset into a dataset of desired format.
3. Add a pointer to this function in the dictionary TASK_TO_REFORMAT_FUN above the load_dataset_by_task function in the same file.
4. Add an entry with the required meta-data for this task in Experiment.py, in the dictionary task_to_info_dict.
5. Add an entry in bert_experiment_driver.py for the corresponding ExperimentBert class
6. The dataset, reformatting, and meta-data loading is done automatically when a child instance of Experiment is used.

• Metric results are stored in json-files with the following path: results/experiments/metrics/<task_name>/<model_name>/metrics.json

• Ray (library for HPS) results are stored in results/ray_results, and can be cleared after the experiments are done. Can be done automatically with clear_result_checkpoints.sh.

• HF Trainer checkpoints are stored in results/trainer_output, and can be cleared after the experiments are done. This is done automatically in the BertExperiment class, and with clear_result_checkpoints.sh

All tasks are evaluated using the Krippendorff Alpha metric (nominal for classification, and interval for regression).

The search space for all tasks and transformer models were the following:

{
    "learning_rate": [1e-5, 2e-5, 3e-5, 4e-5],
    "batch_size": [16, 32]
}

with the exception of SweMNLI, that had its search space reduced to the immense training set size:

{
    "learning_rate": [1e-5, 4e-5],
    "batch_size": [16, 32]
}

Furthermore, all models use the following hyperparameters along with HuggingFace Trainer's default arguments:

{
    "warmup_ratio": 0.06,
    "weight_decay": 0.1,  # 0.0 if gpt
    "num_train_epochs": 10,
    "fp16": true
}

num_train_epochs=10 is the maximum epochs, using early stopping with patience=5.

Below follow the selected Hyperparameters for each model and task, along with a standard deviation of the evaluation metric for the different configurations.

AI-Nordics/bert-large-swedish-cased

KB/bert-base-swedish-cased

KBLab/bert-base-swedish-cased-new

KBLab/megatron-bert-base-swedish-cased-600k

KBLab/megatron-bert-large-swedish-cased-165k

NbAiLab/nb-bert-base

xlm-roberta-base

xlm-roberta-large

The following table shows the average standard deviation of the hyperparameter configuration performances. Sorted on avg std and could indicate hyperparameter sensitivity of the models.

The following table shows the average of the mean distances to the maximum achieved performance. I.e. for each task hyperparameter search, take the mean of the metric distances to the maximum hyperparameter configuration.

The random baseline takes random values from the range of all seen labels, not the current number of possible answers.

Traditional ML baselines take a random answer from the candidates that the models independently predict as a correct answer.

For these traditional ML baselines, only 5% (20,000 samples) of the training set is used for training. This did not seem to have a noticeable effect on the end performance, and the motivation for this was to reduce the training time.