implement test cases for reproducing the results #19
Conversation
elk/tests/test_extraction_output.py
Outdated
| assert row["accuracy"] <= current_performance.iloc[index]["accuracy"] | ||
| assert row["std"] <= current_performance.iloc[index]["std"] |
There was a problem hiding this comment.
I guess we should maybe add some +/- epsilon (e.g having value 3) to increase the tolerance a bit (?)
There was a problem hiding this comment.
Yeah probably. We should make sure the random seed is fixed and then just see empirically how big the discrepancy is with the current code
lauritowal
changed the title
elk/tests/test_extraction_output.py
Outdated
| models_layer_num = default_config["models_layer_num"] | ||
|
|
||
|
|
||
| def train_and_evaluate(model_name, dataset): |
There was a problem hiding this comment.
We want to be able to train on one dataset and evaluate either on the same or another dataset.
elk/tests/test_extraction_output.py
Outdated
| train_and_evaluate(model_name, dataset) | ||
|
|
||
| # Read the contents of the csv files into pandas dataframes | ||
| benchmark_performance = pd.read_csv(f"../test_data/{model_name}_confusion_0.csv") |
There was a problem hiding this comment.
To be able to reproduce the results of the paper, we want mostly the prefix "normal" not "confusion" --> See parser file. However, it might make sense to have an option to be able to select between the different prefixes for the test
elk/tests/test_extraction_output.py
Outdated
|
|
||
| # Compare the contents of the dataframes, | ||
| # make sure current performance is within threshold (epsilon) or better | ||
| epsilon = 3 |
There was a problem hiding this comment.
it would be better if we pass epsilon, instead of hard-coding it here
elk/tests/test_extraction_output.py
Outdated
| Train and evaluate the model with the given dataset. | ||
| """ | ||
| training_command = "python -m elk.train " | ||
| f"--model {model_name} --prefix normal --dataset {dataset} --num_data 1000 --seed 0" |
There was a problem hiding this comment.
Maybe it is better, if we call the functions from train.py and evaluate.py directly?
elk/tests/test_extraction_output.py
Outdated
| train_and_evaluate(model_name, dataset) | ||
|
|
||
| # Read the contents of the csv files into pandas dataframes | ||
| benchmark_performance = pd.read_csv(f"../test_data/{model_name}_confusion_0.csv") |
There was a problem hiding this comment.
You might want to create this csv file (probably manually) and add it to the repo, based on the format / structure of the current .csv there... It should contain all the different combinations from the table in the paper (for logistic regression and CCS only): https://ar5iv.labs.arxiv.org/html/2212.03827/assets/x8.png
only dataframe comparison for now, need to add evaluations