This is a submission of assignment 3 for the CIS711 course.
It contains the code necessary to conduct tests among selected algorithms and determine the most efficient algorithm.
This repository is merely a demonstration of how to perform statistical tests.
Clone the project from GitHub
$ git clone https://github.com/tariqshaban/algorithm-comparator.git
Install numpy
pip install numpy
Install pandas
pip install pandas
Install matplotlib
pip install matplotlib
Install seaborn
pip install seaborn
Install scipy
pip install scipy
Install tabulate
pip install tabulate
Install statsmodels
pip install statsmodels
Install scikit_posthocs
pip install scikit_posthocs
Install deprecation
pip install deprecation
You may need to configure the Python interpreter (depending on the used IDE)
No further configuration is required.
├── README.md <- The top-level README for developers using this project.
│
├── assets
│ │── algorithms <- Storing the provided algorithm's files.
│ │── cached_instances <- Storing processed tables.
│ └── images <- Storing readme image files.
│
├── enums
│ └── adjusted_p_value_methods <- Enumerate adjusted p-value methods.
│
├── helpers
│ │── dataframe_beautifier <- Static methods for beautifying the output of the dataframe.
│ └── progress_handler <- Set of static methods that aid some progress manipulations.
│
├── providers
│ │── data_acquisition_provider <- Static methods which handles data acquisition and tranformation.
│ │── data_manifest_provider <- Static final attributes which informs any functionality of the excepted data to be received.
│ │── plots_provider <- Static methods which perform the plotting functionality.
│ └── non_parametric_tests_provider <- Static methods which handles implementing nonparametric tests the transformed data.
│
└── main <- Acts as a sandbox for methods invocation
Data obtained from the CEC-2014 are stored in assets/algorithms
, it consists of the following:
- A collection of folders, each folder denotes an algorithm.
- Each algorithm contains a collection of textual files, each file denotes a pair of problem/dimension
- Each problem/dimension pair contains a table, columns denote the iteration and rows denote the parameter
- Each cell denotes the difference between the actual and the optimal result for the respected algorithm/dimension/problem/parameter/iteration (the less the value is, the better)
Refer to the following tree for a better understanding
└── CEC-2014
│── b3e3pbest
│ │── 10D
│ │ │── problem 1
│ │ │ │── parameter 1
│ │ │ │ │── iteration 1
│ │ │ │ │── iteration 2
│ │ │ │ │── ...
│ │ │ │ └── iteration 51
│ │ │ │── parameter 2
│ │ │ │── ...
│ │ │ └── parameter 14
│ │ │── problem 2
│ │ │── problem 3
│ │ │── ...
│ │ └── problem 30
│ │── 30D
│ │── 50D
│ └── 100D
│── CMLSP
│── DE_b6e6rlwithrestart
│── ...
└── UMOEAS
If you wish to run the program and show results based on your own data, you should perform the following steps:
- Ensure that your input values has the exact same structure as in the provided
assets/algorithms
folder, by achieving the following steps:- The dimensions must be within [10, 30, 50, 100]. If need be, modify the
providers/data_manifest_provider.py
DIMENSIONS
variable - The parameters must be exactly 14. If need be, modify the
providers/data_manifest_provider.py
PARAMETERS
variable - The folders must follow the exact structural requirement as follows:
- List of folders (denoting the algorithms)
- These folders contain text files that follow this format:
ALGO-NAME_PROBLEM_DIMENSION
- These files contain a tabular form denoting iterations as columns and the parameters as the rows
- The dimensions must be within [10, 30, 50, 100]. If need be, modify the
- Call
DataAcquisitionProvider.set_algorithms_raw_directory('MY_PATH')
from themain.py
- Delete the contents of the
assets/cached_instances
folder to prepare for a new cached snapshot (Optional - Apply when elevated permissions are required) - Call
DataAcquisitionProvider.cache_algorithms_comparisons()
frommain.py
- Verify that the cached instances are generated, by observing
assets/cached_insances
Note: The program will not function if you delete the assets/cached_instances
folder without providing a proper
snapshot (you must call DataAcquisitionProvider.cache_algorithms_comparisons()
from main.py
before invoking any
method to automatically generate such snapshot)
Data received from the CEC-2014 contains unnecessary additional files which should be removed.
Raw data from the textual files did not seem to require any preprocessing, since the data is assumed to be correct, e.g. you cannot confirm that a difference of 3.2987*10^8 is considered as an entry error or that the algorithm did actually obtain this result under certain circumstances. The only preprocessing that could have been done is to remove any non-positive values, but there was none already.
Normality check
![]()
It can be clearly indicated from this figure that the distribution of this algorithm is not normal, hence we should conduct non-parametric tests.
![]()
The qq plot (which shows the distribution of the data against the expected normal distribution), should have the observations lie approximately on a straight line to consider the distribution normal. The figure also indicate that this not a normal distribution.
Comparisons
![]()
For a given dimension and parameter, we can estimate the best algorithm by using the mean value, in this plot, we can conclude that UMOEAS is the best algorithm, since it has the least difference to the optimal result. However, the best algorithm might be different for each dimension and parameter.
![]()
To further find the outliers in each algorithm distribution, we used the box plot, having a whisker which converges to the zero highly indicate that such algorithm might be the best. The best algorithm usually doesn't have outliers, or it is not very distant from the maximum whisker, we can estimate from the plot that UMOEAS is most likely the best algorithm since it has the lowest median (central horizontal line in the box), as well as the Q1 is relatively low.
Overall Performance
![]()
Display the performance for algorithm for a given parameter while escalating the dimensionality, some algorithms surprisingly obtain better results when increasing the dimensionality, perhaps the algorithm is optimized to run on a higher, demanding dimensions?
![]()
The previous figure has been normalized to further facilitate the interpretation of the performance, notice that some algorithms doesn't have any performance indicators at 100D, this is because that the raw data didn't record any observation that correlated the algorithm with the dimension.
The first step was to load the raw txt file into a nested dictionary and tag the required information as the key, while the textual content as the value in a dataframe format.
Then nested dictionary was refactored to the following manner:
- A dictionary consisting of:
- Key: Dimension name
- Value: A dictionary consisting of:
- Key: Parameter name
- Value: A dataframe consisting of:
- Columns: Algorithm Name
- Row: Problem number
- Cell: Consists of mean and standard deviation of the provided 51 iteration
Note that the parameters follow zero based indexing, 0 denotes the first line in the txt file.
This ensures that the data is suitable for comparisons and tests, standard deviation is nullified if not needed.
The inspections above became possible after this step.
Identifying the algorithm with the highest wins and ties can give a broad indication that this might be the best algorithm.
This is calculated by using the following procedure:
- Identifying the least mean in each problem (DEPRECATED)
- Applying Mann–Whitney U test for each entry by comparing the raw iterations, note that:
- Caching will not be beneficial since we resort to the raw iterations rather than the mean
- Wilcoxon will not function since it appears that some algorithms recorded fewer iterations (sample size is not identical)
- Mann–Whitney should be used over Wilcoxon for independent variables (which is true in this case)
Algorithm comparison with w/t/l (deprecated)
Dimension: 10
Parameter: 8
(Problem,'Mean/Std') | b3e3pbest | CMLSP | DE_b6e6rlwithrestart | FCDE | FERDE | FWA-DM | GaAPADE | L-SHADE | MVMO | NRGA | OptBees | POBL_ADE | rmalschcma | RSDE | SOO | SOO+BOBYQA | UMOEAS |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
(1, 'Mean') | 3.2173e+06 | 2.3678e-04 | 5.8170e-10 | 3.0817e+05 | 5.5689e+06 | 1.4336e+04 | 1.3320e-10 | 0.0000e+00 | 5.8577e+01 | 8.0011e+04 | 1.4400e+04 | 8.9204e+04 | 0.0000e+00 | 1.8677e-07 | 8.8108e+06 | 4.3670e+04 | 0.0000e+00 |
(1, 'Std') | 9.1182e+06 | 7.4361e-04 | 3.0164e-09 | 8.2251e+05 | 1.1186e+07 | 2.2851e+04 | 4.8530e-10 | 0.0000e+00 | 1.3415e+02 | 1.2553e+05 | 1.9735e+04 | 3.4845e+05 | 0.0000e+00 | 8.2445e-07 | 0.0000e+00 | 2.2668e-11 | 0.0000e+00 |
(2, 'Mean') | 1.3453e+08 | 1.1146e-15 | 0.0000e+00 | 4.2266e+07 | 3.1485e+08 | 1.4125e-04 | 5.7284e-10 | 0.0000e+00 | 2.7764e-05 | 9.2183e+02 | 1.2668e+03 | 1.7441e+05 | 0.0000e+00 | 0.0000e+00 | 1.0782e+01 | 4.2900e-01 | 2.6778e-06 |
(2, 'Std') | 4.2102e+08 | 5.5169e-15 | 0.0000e+00 | 1.5111e+08 | 9.1609e+08 | 9.5932e-04 | 3.3016e-09 | 0.0000e+00 | 4.4078e-05 | 1.1044e+03 | 1.4341e+03 | 1.1391e+06 | 0.0000e+00 | 0.0000e+00 | 1.9267e-15 | 0.0000e+00 | 1.7449e-05 |
(3, 'Mean') | 3.1643e+03 | 2.1990e-03 | 0.0000e+00 | 4.7480e+02 | 3.4392e+03 | 6.1074e-07 | 3.5666e-14 | 0.0000e+00 | 1.7640e-08 | 1.8000e+03 | 4.4148e+01 | 1.9483e-02 | 9.7943e+01 | 0.0000e+00 | 6.6483e+03 | 6.3311e+03 | 0.0000e+00 |
(3, 'Std') | 7.8280e+03 | 4.4071e-03 | 0.0000e+00 | 1.3640e+03 | 7.0176e+03 | 3.3466e-06 | 1.4932e-13 | 0.0000e+00 | 6.9825e-08 | 1.5625e+03 | 1.1830e+02 | 6.0747e-02 | 3.3176e+02 | 0.0000e+00 | 1.8234e-12 | 5.0942e-13 | 0.0000e+00 |
(4, 'Mean') | 2.7916e+01 | 4.4583e-15 | 6.6299e+00 | 5.7926e+00 | 3.0825e+01 | 2.5263e+00 | 3.0688e+01 | 2.9410e+01 | 9.5457e+00 | 1.5468e+01 | 3.2050e+00 | 2.9914e+01 | 8.5008e-02 | 1.2103e+01 | 1.5630e+00 | 0.0000e+00 | 1.1028e-03 |
(4, 'Std') | 5.1019e+01 | 1.5282e-14 | 1.2494e+01 | 1.3499e+01 | 6.7031e+01 | 1.5072e+00 | 1.1206e+01 | 1.2464e+01 | 1.4838e+01 | 1.6855e+01 | 2.9591e+00 | 1.4841e+01 | 6.0110e-01 | 1.6092e+01 | 2.5450e-16 | 0.0000e+00 | 6.8692e-03 |
(5, 'Mean') | 2.0224e+01 | 1.8898e+01 | 1.9249e+01 | 2.0488e+01 | 2.0102e+01 | 2.0117e+01 | 1.9813e+01 | 1.9093e+01 | 1.7366e+01 | 1.9607e+01 | 2.0000e+01 | 1.9812e+01 | 1.4286e+01 | 2.0001e+01 | 2.0000e+01 | 2.0000e+01 | 1.8844e+01 |
(5, 'Std') | 1.7312e-01 | 4.4276e+00 | 2.6751e+00 | 1.6668e-01 | 1.8721e-01 | 7.5261e-02 | 1.0909e+00 | 2.6043e+00 | 6.3759e+00 | 2.7728e+00 | 1.2727e-04 | 1.9046e+00 | 9.2228e+00 | 4.2556e-03 | 0.0000e+00 | 0.0000e+00 | 4.6818e+00 |
(6, 'Mean') | 3.3843e+00 | 5.0749e-01 | 1.7540e-02 | 5.7703e+00 | 3.6179e+00 | 7.2268e-01 | 5.1955e-01 | 5.5957e-02 | 1.6387e-02 | 2.5269e+00 | 3.8299e+00 | 1.0395e+00 | 1.7455e-03 | 3.6645e-01 | 2.0000e-03 | 2.0000e-03 | 1.1453e-03 |
(6, 'Std') | 3.9860e+00 | 1.0751e+00 | 1.2403e-01 | 1.3040e+00 | 3.3658e+00 | 6.4668e-01 | 9.0086e-01 | 2.0668e-01 | 4.3072e-02 | 1.2781e+00 | 1.6062e+00 | 7.7488e-01 | 6.0439e-03 | 6.9354e-01 | 3.0364e-19 | 3.0364e-19 | 7.9660e-03 |
(7, 'Mean') | 3.7232e+00 | 0.0000e+00 | 4.6975e-02 | 6.0965e-01 | 5.8060e+00 | 9.4823e-02 | 2.2279e-02 | 3.8303e-02 | 1.8855e-02 | 2.2108e-01 | 1.5978e-01 | 1.7237e-01 | 1.0379e-05 | 6.2573e-02 | 4.9000e-02 | 4.9000e-02 | 0.0000e+00 |
(7, 'Std') | 1.0662e+01 | 0.0000e+00 | 1.8042e-02 | 1.3960e+00 | 1.4586e+01 | 4.9192e-02 | 1.8054e-02 | 2.7094e-02 | 1.4428e-02 | 1.1435e-01 | 1.4321e-01 | 1.9425e-01 | 7.3392e-05 | 3.6570e-02 | 2.9149e-18 | 2.9149e-18 | 0.0000e+00 |
(8, 'Mean') | 1.4194e+01 | 4.2705e+00 | 0.0000e+00 | 1.6896e+01 | 8.8301e+00 | 2.5362e-01 | 1.9352e-01 | 2.4588e-02 | 3.2325e-03 | 7.2560e+00 | 6.6335e-01 | 8.0704e+00 | 2.3517e+00 | 5.7989e+00 | 2.0894e+01 | 2.0894e+01 | 1.4714e-03 |
(8, 'Std') | 2.2096e+01 | 6.1009e+00 | 0.0000e+00 | 7.1114e+00 | 1.8641e+01 | 8.0862e-01 | 1.4386e-01 | 5.5851e-02 | 1.7291e-02 | 4.7402e+00 | 8.7374e-01 | 3.7321e+00 | 1.1196e+00 | 3.2373e+00 | 3.5527e-15 | 3.5527e-15 | 1.0404e-02 |
(9, 'Mean') | 2.3721e+01 | 2.6638e+00 | 7.3505e+00 | 3.1234e+01 | 2.4816e+01 | 7.3757e+00 | 7.1308e+00 | 7.0373e+00 | 4.0650e+00 | 8.7403e+00 | 2.2825e+01 | 8.4680e+00 | 3.5117e+00 | 1.1570e+01 | 9.9500e+00 | 9.9500e+00 | 2.9820e+00 |
(9, 'Std') | 2.1257e+01 | 1.5192e+00 | 1.2518e+00 | 1.3436e+01 | 2.2855e+01 | 2.9762e+00 | 1.7279e+00 | 1.3367e+00 | 1.4060e+00 | 3.9125e+00 | 8.0383e+00 | 4.3731e+00 | 1.5928e+00 | 5.4159e+00 | 1.9267e-15 | 1.9267e-15 | 2.6317e+00 |
(10, 'Mean') | 3.9466e+02 | 3.4046e+02 | 4.8984e-02 | 2.5025e+02 | 1.9362e+02 | 1.5927e+00 | 1.3860e+01 | 1.2554e+01 | 1.5078e+01 | 1.5601e+02 | 2.7223e+02 | 1.6802e+02 | 7.7705e+00 | 1.7084e+02 | 1.3046e+02 | 1.3046e+02 | 3.9117e-01 |
(10, 'Std') | 6.1160e+02 | 1.8750e+02 | 6.6962e-02 | 4.5372e+02 | 3.9711e+02 | 2.0752e+00 | 4.9536e+00 | 3.8685e+00 | 5.9351e+00 | 1.3404e+02 | 1.0902e+02 | 1.1596e+02 | 2.3231e+01 | 1.3911e+02 | 0.0000e+00 | 0.0000e+00 | 7.8806e-01 |
(11, 'Mean') | 9.4881e+02 | 3.1649e+02 | 3.3836e+02 | 9.4499e+02 | 6.0638e+02 | 4.9096e+02 | 4.0565e+02 | 3.1326e+02 | 1.3377e+02 | 6.3858e+02 | 5.8169e+02 | 4.0679e+02 | 3.5985e+01 | 5.1360e+02 | 3.5616e+02 | 5.7042e+02 | 2.4513e+02 |
(11, 'Std') | 5.0775e+02 | 2.0601e+02 | 9.3031e+01 | 5.7735e+02 | 5.8480e+02 | 1.9147e+02 | 1.4193e+02 | 1.1954e+02 | 8.2245e+01 | 3.2230e+02 | 2.1961e+02 | 2.0849e+02 | 5.7824e+01 | 3.1555e+02 | 5.7398e-14 | 1.3878e-13 | 2.0285e+02 |
(12, 'Mean') | 8.6668e-01 | 3.0265e-02 | 4.0413e-01 | 8.5524e-01 | 7.3229e-01 | 1.5814e-01 | 2.7605e-01 | 4.0242e-01 | 1.4101e-01 | 1.3621e-01 | 2.0598e-01 | 4.1926e-01 | 2.5293e-02 | 2.7099e-01 | 5.0000e-02 | 5.0000e-02 | 0.0000e+00 |
(12, 'Std') | 6.7771e-01 | 7.1535e-02 | 8.8846e-02 | 6.1100e-01 | 5.3903e-01 | 1.3866e-01 | 5.6198e-02 | 6.4981e-02 | 4.0987e-02 | 8.9103e-02 | 1.1115e-01 | 9.3798e-02 | 3.4350e-02 | 1.8276e-01 | 2.9149e-18 | 2.9149e-18 | 0.0000e+00 |
(13, 'Mean') | 3.4841e-01 | 2.8535e-02 | 1.5419e-01 | 2.5917e-01 | 3.7657e-01 | 1.4952e-01 | 7.4203e-02 | 1.0130e-01 | 8.8528e-02 | 1.6061e-01 | 4.2972e-01 | 1.3876e-01 | 6.1959e-02 | 1.3908e-01 | 3.0000e-02 | 3.0000e-02 | 2.7973e-02 |
(13, 'Std') | 3.3201e-01 | 1.3624e-02 | 2.9855e-02 | 8.7584e-02 | 5.4088e-01 | 7.5435e-02 | 1.3705e-02 | 1.8868e-02 | 2.3427e-02 | 6.2773e-02 | 1.9940e-01 | 4.9659e-02 | 2.3682e-02 | 3.9528e-02 | 6.9559e-18 | 6.9559e-18 | 1.1768e-02 |
(14, 'Mean') | 7.7268e-01 | 2.3726e-01 | 1.4813e-01 | 2.7023e-01 | 6.1937e-01 | 2.3112e-01 | 1.0025e-01 | 1.0968e-01 | 1.5941e-01 | 2.8415e-01 | 4.5348e-01 | 2.7379e-01 | 1.6997e-01 | 1.5763e-01 | 1.6000e-01 | 1.5000e-01 | 1.3726e-01 |
(14, 'Std') | 2.2837e+00 | 6.1505e-02 | 3.7442e-02 | 4.0008e-01 | 1.2697e+00 | 1.1931e-01 | 3.1699e-02 | 2.9878e-02 | 4.2428e-02 | 7.1869e-02 | 2.8332e-01 | 1.2089e-01 | 4.7223e-02 | 5.1255e-02 | 7.7731e-18 | 2.7756e-17 | 4.0418e-02 |
(15, 'Mean') | 7.9821e+01 | 9.0827e-01 | 1.0463e+00 | 2.1662e+00 | 1.0522e+01 | 1.0616e+00 | 9.0424e-01 | 8.8466e-01 | 7.3805e-01 | 1.0853e+00 | 2.7050e+00 | 8.6952e-01 | 6.4712e-01 | 9.9273e-01 | 4.4000e-01 | 4.2000e-01 | 8.0506e-01 |
(15, 'Std') | 3.2165e+02 | 2.3424e-01 | 1.9880e-01 | 1.5579e+00 | 1.9386e+01 | 3.6908e-01 | 1.5625e-01 | 1.4907e-01 | 1.7228e-01 | 5.3661e-01 | 1.2621e+00 | 2.6713e-01 | 1.8934e-01 | 3.7081e-01 | 6.3626e-17 | 1.1129e-16 | 2.1669e-01 |
(16, 'Mean') | 2.7971e+00 | 2.4018e+00 | 2.2131e+00 | 3.4914e+00 | 2.3434e+00 | 1.9837e+00 | 2.3152e+00 | 2.1553e+00 | 1.9729e+00 | 2.8734e+00 | 2.8781e+00 | 1.7489e+00 | 1.9725e+00 | 2.3899e+00 | 2.5200e+00 | 2.5200e+00 | 2.3951e+00 |
(16, 'Std') | 6.6579e-01 | 8.0481e-01 | 2.4108e-01 | 1.5364e-01 | 8.6284e-01 | 5.0964e-01 | 2.9011e-01 | 2.4824e-01 | 2.7102e-01 | 4.3232e-01 | 3.9732e-01 | 4.9581e-01 | 4.8523e-01 | 4.8570e-01 | 1.8655e-16 | 1.8655e-16 | 3.6969e-01 |
(17, 'Mean') | 2.7851e+04 | 3.1275e+02 | 1.3597e+01 | 4.0135e+03 | 4.4581e+05 | 2.5879e+02 | 3.8214e+01 | 2.9194e+01 | 1.8521e+01 | 5.2643e+04 | 1.9985e+03 | 2.5739e+02 | 1.2158e+02 | 9.7188e+01 | 3.1229e+06 | 1.5095e+03 | 1.8386e+01 |
(17, 'Std') | 9.1488e+04 | 1.6220e+02 | 3.2419e+01 | 1.2894e+04 | 9.3368e+05 | 1.7635e+02 | 2.0012e+01 | 1.3779e+01 | 1.6172e+01 | 5.5611e+04 | 2.2935e+03 | 1.6174e+02 | 1.3055e+02 | 7.5332e+01 | 0.0000e+00 | 1.2735e-13 | 2.0521e+01 |
(18, 'Mean') | 3.1651e+05 | 3.4780e+01 | 7.2311e-01 | 6.8721e+02 | 6.7813e+05 | 2.5187e+01 | 1.7535e+00 | 1.5763e+00 | 8.5047e-01 | 7.4526e+03 | 1.0485e+03 | 3.3189e+01 | 2.8066e+01 | 3.8724e+00 | 1.2933e+04 | 1.0125e+04 | 9.7137e-01 |
(18, 'Std') | 1.5418e+06 | 1.6790e+01 | 6.3765e-01 | 2.3847e+03 | 2.3833e+06 | 1.8260e+01 | 4.9605e-01 | 6.3699e-01 | 7.6551e-01 | 5.0623e+03 | 2.3941e+03 | 3.3291e+01 | 2.4611e+01 | 3.6269e+00 | 2.3758e-12 | 2.0849e-12 | 7.0514e-01 |
(19, 'Mean') | 2.6420e+00 | 1.3411e+00 | 3.2565e-01 | 4.0970e+00 | 2.2598e+00 | 1.3184e+00 | 7.7791e-01 | 8.6626e-01 | 4.1739e-01 | 2.1745e+00 | 1.1459e+00 | 2.2578e+00 | 2.2829e-01 | 1.6202e+00 | 5.5000e-01 | 5.5000e-01 | 7.7917e-01 |
(19, 'Std') | 3.4365e+00 | 6.1747e-01 | 1.3382e-01 | 5.4868e-01 | 2.7350e+00 | 7.7368e-01 | 2.3866e-01 | 2.7106e-01 | 2.4533e-01 | 7.7558e-01 | 4.5871e-01 | 1.1836e+00 | 1.0815e-01 | 6.6285e-01 | 4.6639e-17 | 4.6639e-17 | 3.0997e-01 |
(20, 'Mean') | 1.1091e+04 | 2.4226e+01 | 2.5344e-01 | 1.5224e+02 | 8.3414e+03 | 1.3382e+01 | 1.2239e+00 | 1.3850e+00 | 4.3605e-01 | 3.0909e+03 | 1.7151e+01 | 1.2599e+01 | 4.1577e+01 | 1.0938e+00 | 9.3693e+03 | 8.8034e+03 | 7.0498e-01 |
(20, 'Std') | 4.2736e+04 | 1.3454e+01 | 1.5281e-01 | 4.6206e+02 | 1.9270e+04 | 1.1629e+01 | 2.2908e-01 | 3.4465e-01 | 4.2103e-01 | 2.9752e+03 | 3.2422e+01 | 1.1634e+01 | 4.7999e+01 | 6.8761e-01 | 1.0188e-12 | 1.9730e-12 | 3.7165e-01 |
(21, 'Mean') | 1.7722e+04 | 4.6631e+01 | 1.5300e+00 | 4.2818e+03 | 7.2031e+04 | 9.4695e+01 | 3.7143e+00 | 2.2426e+00 | 2.1863e+00 | 5.2251e+03 | 1.1665e+02 | 1.0997e+02 | 2.0352e+02 | 1.1406e+01 | 1.5952e+07 | 2.0929e+03 | 9.8912e-01 |
(21, 'Std') | 7.3593e+04 | 5.6676e+01 | 3.9426e+00 | 1.0291e+04 | 1.7250e+05 | 9.7934e+01 | 2.0267e+00 | 1.1156e+00 | 4.8888e+00 | 4.5464e+03 | 1.1451e+02 | 1.1875e+02 | 2.0210e+02 | 2.7322e+01 | 0.0000e+00 | 4.9324e-13 | 3.8746e-01 |
(22, 'Mean') | 7.9003e+01 | 1.1984e+02 | 4.0742e-01 | 4.2709e+01 | 4.5660e+01 | 3.4109e+01 | 1.6056e+01 | 1.1004e+01 | 2.0681e+00 | 3.7947e+01 | 1.9069e+01 | 3.0770e+01 | 1.3125e+01 | 1.8241e+01 | 1.2790e+02 | 1.2862e+02 | 4.7301e+00 |
(22, 'Std') | 1.4114e+02 | 4.6158e+01 | 8.1922e-01 | 8.0402e+01 | 9.1077e+01 | 4.4019e+01 | 3.3214e+00 | 2.9481e+00 | 8.3261e-01 | 3.9524e+01 | 8.5348e+00 | 3.3428e+01 | 1.6994e+01 | 7.3673e+00 | 4.3369e-14 | 1.1940e-14 | 4.7595e+00 |
(23, 'Mean') | 3.3407e+02 | 2.0214e+02 | 3.2946e+02 | 3.2986e+02 | 3.3411e+02 | 3.2946e+02 | 3.2946e+02 | 3.2946e+02 | 3.2946e+02 | 3.2946e+02 | 3.2300e+02 | 3.2956e+02 | 3.2946e+02 | 3.2946e+02 | 2.0000e+02 | 2.0000e+02 | 3.2946e+02 |
(23, 'Std') | 1.3329e+01 | 1.5165e+01 | 6.5153e-14 | 1.3651e+00 | 1.1698e+01 | 1.7373e-07 | 5.5718e-14 | 7.4243e-14 | 6.9391e-14 | 3.3440e-05 | 4.5679e+01 | 3.6458e-01 | 6.5153e-14 | 6.9391e-14 | 0.0000e+00 | 0.0000e+00 | 6.9391e-14 |
(24, 'Mean') | 1.3208e+02 | 1.5608e+02 | 1.1504e+02 | 1.4048e+02 | 1.3620e+02 | 1.2757e+02 | 1.1204e+02 | 1.1220e+02 | 1.0923e+02 | 1.3251e+02 | 1.3868e+02 | 1.2406e+02 | 1.0855e+02 | 1.2089e+02 | 1.1744e+02 | 1.1744e+02 | 1.0846e+02 |
(24, 'Std') | 2.4287e+01 | 3.8454e+01 | 2.5641e+00 | 8.0820e+00 | 3.0184e+01 | 2.8895e+01 | 1.7669e+00 | 1.8798e+00 | 2.9471e+00 | 1.7014e+01 | 1.1506e+01 | 2.3796e+01 | 2.9210e+00 | 7.3760e+00 | 1.8561e-14 | 1.8561e-14 | 2.6853e+00 |
(25, 'Mean') | 1.7666e+02 | 1.5905e+02 | 1.3970e+02 | 1.7162e+02 | 1.7090e+02 | 1.7970e+02 | 1.6729e+02 | 1.4058e+02 | 1.1613e+02 | 1.8403e+02 | 1.4973e+02 | 1.8632e+02 | 1.7606e+02 | 1.4864e+02 | 1.4781e+02 | 1.4524e+02 | 1.3083e+02 |
(25, 'Std') | 3.4713e+01 | 3.0566e+01 | 2.9976e+01 | 3.6765e+01 | 1.7934e+01 | 2.7868e+01 | 3.7162e+01 | 3.5406e+01 | 7.2308e+00 | 2.0448e+01 | 1.5719e+01 | 2.6080e+01 | 3.1009e+01 | 3.1884e+01 | 2.3879e-14 | 3.2576e-14 | 1.8335e+01 |
(26, 'Mean') | 1.0038e+02 | 1.0003e+02 | 1.0015e+02 | 1.0027e+02 | 1.0041e+02 | 1.0017e+02 | 1.0008e+02 | 1.0010e+02 | 1.0009e+02 | 1.0014e+02 | 1.0041e+02 | 1.0013e+02 | 1.0005e+02 | 1.0013e+02 | 1.0005e+02 | 1.0005e+02 | 1.0003e+02 |
(26, 'Std') | 4.4180e-01 | 3.0350e-02 | 3.6458e-02 | 5.0440e-02 | 5.7917e-01 | 7.3983e-02 | 1.5571e-02 | 1.9919e-02 | 2.2825e-02 | 6.2282e-02 | 1.8620e-01 | 5.1664e-02 | 2.0154e-02 | 4.0780e-02 | 1.8561e-14 | 1.8561e-14 | 2.0252e-02 |
(27, 'Mean') | 1.4929e+02 | 1.8371e+02 | 1.1107e+02 | 3.5280e+02 | 3.8543e+02 | 3.2132e+02 | 8.9778e+01 | 5.8410e+01 | 1.7852e+01 | 2.8152e+02 | 9.1119e+00 | 2.5600e+02 | 1.8598e+02 | 1.6910e+02 | 2.0000e+02 | 2.0000e+02 | 2.5705e+01 |
(27, 'Std') | 1.7578e+02 | 1.2663e+02 | 1.6004e+02 | 1.1954e+02 | 5.3540e+01 | 1.2062e+02 | 1.5966e+02 | 1.3262e+02 | 7.7221e+01 | 1.5528e+02 | 2.9681e+00 | 1.6633e+02 | 1.5509e+02 | 1.5883e+02 | 0.0000e+00 | 0.0000e+00 | 8.0055e+01 |
(28, 'Mean') | 4.3055e+02 | 2.9229e+02 | 3.7206e+02 | 7.3672e+02 | 4.5149e+02 | 3.4854e+02 | 3.8321e+02 | 3.8081e+02 | 3.6154e+02 | 4.8204e+02 | 3.0671e+02 | 4.2341e+02 | 3.8885e+02 | 4.0599e+02 | 2.0000e+02 | 2.0000e+02 | 3.1292e+02 |
(28, 'Std') | 1.1046e+02 | 7.8891e+01 | 2.7534e+01 | 2.7323e+01 | 1.3755e+02 | 4.7819e+01 | 3.0919e+01 | 3.1362e+01 | 4.2393e+01 | 1.0746e+02 | 2.4539e-01 | 5.5257e+01 | 8.0680e+01 | 6.1578e+01 | 0.0000e+00 | 0.0000e+00 | 1.8413e+01 |
(29, 'Mean') | 1.2941e+05 | 2.0000e+02 | 2.1815e+02 | 3.1238e+02 | 4.7797e+04 | 2.1175e+02 | 2.2242e+02 | 2.2212e+02 | 1.8295e+02 | 4.1909e+02 | 2.2139e+02 | 3.5544e+05 | 2.3243e+02 | 3.4030e+04 | 2.0000e+02 | 2.0000e+02 | 1.9812e+02 |
(29, 'Std') | 6.2040e+05 | 0.0000e+00 | 2.0262e+01 | 1.9044e+02 | 1.7040e+05 | 2.0789e+01 | 6.8231e-01 | 4.5851e-01 | 3.7331e+01 | 7.3024e+01 | 2.2553e+01 | 9.3097e+05 | 1.1241e+01 | 2.3907e+05 | 0.0000e+00 | 0.0000e+00 | 1.1791e+01 |
(30, 'Mean') | 1.6476e+03 | 2.1687e+02 | 4.7501e+02 | 9.4760e+02 | 1.6034e+03 | 3.9582e+02 | 4.6841e+02 | 4.6538e+02 | 5.0008e+02 | 1.7725e+03 | 3.9700e+02 | 6.3914e+02 | 6.2537e+02 | 5.5999e+02 | 2.0000e+02 | 2.0000e+02 | 2.3971e+02 |
(30, 'Std') | 2.6779e+03 | 3.9449e+01 | 2.4295e+01 | 8.8907e+02 | 2.0785e+03 | 1.1722e+02 | 1.8729e+01 | 1.3231e+01 | 2.6626e+01 | 3.0944e+02 | 8.1181e+01 | 1.6204e+02 | 1.1355e+02 | 1.3834e+02 | 0.0000e+00 | 0.0000e+00 | 3.5530e+01 |
w/t/l | 0/4/26 | 4/7/19 | 9/5/16 | 0/2/28 | 0/4/26 | 1/1/28 | 2/3/25 | 3/4/23 | 7/6/17 | 0/0/30 | 2/1/27 | 1/2/27 | 5/6/19 | 1/4/25 | 5/1/24 | 6/1/23 | 0/30/0 |
Algorithm comparison with w/t/l
Dimension: 10
Parameter: 8
Problem | b3e3pbest | CMLSP | DE_b6e6rlwithrestart | FCDE | FERDE | FWA-DM | GaAPADE | L-SHADE | MVMO | NRGA | OptBees | POBL_ADE | rmalschcma | RSDE | SOO | SOO+BOBYQA | UMOEAS |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 3.2173e+06 (l) | 2.3678e-04 (l) | 5.8170e-10 (t) | 3.0817e+05 (l) | 5.5689e+06 (l) | 1.4336e+04 (l) | 1.3320e-10 (l) | 0.0000e+00 (t) | 5.8577e+01 (l) | 8.0011e+04 (l) | 1.4400e+04 (l) | 8.9204e+04 (l) | 0.0000e+00 (t) | 1.8677e-07 (l) | 8.8108e+06 (l) | 4.3670e+04 (l) | 0.0000e+00 (t) |
2 | 1.3453e+08 (w) | 1.1146e-15 (t) | 0.0000e+00 (t) | 4.2266e+07 (l) | 3.1485e+08 (l) | 1.4125e-04 (w) | 5.7284e-10 (l) | 0.0000e+00 (t) | 2.7764e-05 (l) | 9.2183e+02 (l) | 1.2668e+03 (w) | 1.7441e+05 (w) | 0.0000e+00 (t) | 0.0000e+00 (t) | 1.0782e+01 (w) | 4.2900e-01 (w) | 2.6778e-06 (t) |
3 | 3.1643e+03 (l) | 2.1990e-03 (l) | 0.0000e+00 (t) | 4.7480e+02 (l) | 3.4392e+03 (l) | 6.1074e-07 (l) | 3.5666e-14 (l) | 0.0000e+00 (t) | 1.7640e-08 (l) | 1.8000e+03 (l) | 4.4148e+01 (l) | 1.9483e-02 (l) | 9.7943e+01 (l) | 0.0000e+00 (t) | 6.6483e+03 (l) | 6.3311e+03 (l) | 0.0000e+00 (t) |
4 | 2.7916e+01 (l) | 4.4583e-15 (l) | 6.6299e+00 (l) | 5.7926e+00 (t) | 3.0825e+01 (t) | 2.5263e+00 (l) | 3.0688e+01 (l) | 2.9410e+01 (l) | 9.5457e+00 (l) | 1.5468e+01 (l) | 3.2050e+00 (l) | 2.9914e+01 (l) | 8.5008e-02 (l) | 1.2103e+01 (l) | 1.5630e+00 (l) | 0.0000e+00 (w) | 1.1028e-03 (t) |
5 | 2.0224e+01 (l) | 1.8898e+01 (l) | 1.9249e+01 (l) | 2.0488e+01 (l) | 2.0102e+01 (t) | 2.0117e+01 (t) | 1.9813e+01 (l) | 1.9093e+01 (l) | 1.7366e+01 (w) | 1.9607e+01 (l) | 2.0000e+01 (l) | 1.9812e+01 (t) | 1.4286e+01 (t) | 2.0001e+01 (l) | 2.0000e+01 (l) | 2.0000e+01 (l) | 1.8844e+01 (t) |
6 | 3.3843e+00 (l) | 5.0749e-01 (l) | 1.7540e-02 (t) | 5.7703e+00 (l) | 3.6179e+00 (l) | 7.2268e-01 (l) | 5.1955e-01 (l) | 5.5957e-02 (l) | 1.6387e-02 (l) | 2.5269e+00 (l) | 3.8299e+00 (l) | 1.0395e+00 (l) | 1.7455e-03 (l) | 3.6645e-01 (l) | 2.0000e-03 (l) | 2.0000e-03 (l) | 1.1453e-03 (t) |
7 | 3.7232e+00 (l) | 0.0000e+00 (t) | 4.6975e-02 (l) | 6.0965e-01 (l) | 5.8060e+00 (l) | 9.4823e-02 (l) | 2.2279e-02 (l) | 3.8303e-02 (l) | 1.8855e-02 (l) | 2.2108e-01 (l) | 1.5978e-01 (l) | 1.7237e-01 (l) | 1.0379e-05 (t) | 6.2573e-02 (l) | 4.9000e-02 (l) | 4.9000e-02 (l) | 0.0000e+00 (t) |
8 | 1.4194e+01 (l) | 4.2705e+00 (l) | 0.0000e+00 (t) | 1.6896e+01 (l) | 8.8301e+00 (l) | 2.5362e-01 (l) | 1.9352e-01 (l) | 2.4588e-02 (l) | 3.2325e-03 (l) | 7.2560e+00 (l) | 6.6335e-01 (l) | 8.0704e+00 (l) | 2.3517e+00 (l) | 5.7989e+00 (l) | 2.0894e+01 (l) | 2.0894e+01 (l) | 1.4714e-03 (t) |
9 | 2.3721e+01 (l) | 2.6638e+00 (t) | 7.3505e+00 (l) | 3.1234e+01 (l) | 2.4816e+01 (l) | 7.3757e+00 (l) | 7.1308e+00 (l) | 7.0373e+00 (l) | 4.0650e+00 (l) | 8.7403e+00 (l) | 2.2825e+01 (l) | 8.4680e+00 (l) | 3.5117e+00 (t) | 1.1570e+01 (w) | 9.9500e+00 (l) | 9.9500e+00 (l) | 2.9820e+00 (t) |
10 | 3.9466e+02 (l) | 3.4046e+02 (l) | 4.8984e-02 (w) | 2.5025e+02 (l) | 1.9362e+02 (t) | 1.5927e+00 (l) | 1.3860e+01 (l) | 1.2554e+01 (l) | 1.5078e+01 (l) | 1.5601e+02 (l) | 2.7223e+02 (l) | 1.6802e+02 (l) | 7.7705e+00 (l) | 1.7084e+02 (l) | 1.3046e+02 (l) | 1.3046e+02 (l) | 3.9117e-01 (t) |
11 | 9.4881e+02 (l) | 3.1649e+02 (t) | 3.3836e+02 (l) | 9.4499e+02 (l) | 6.0638e+02 (l) | 4.9096e+02 (l) | 4.0565e+02 (l) | 3.1326e+02 (l) | 1.3377e+02 (w) | 6.3858e+02 (l) | 5.8169e+02 (l) | 4.0679e+02 (l) | 3.5985e+01 (l) | 5.1360e+02 (l) | 3.5616e+02 (l) | 5.7042e+02 (l) | 2.4513e+02 (t) |
12 | 8.6668e-01 (l) | 3.0265e-02 (l) | 4.0413e-01 (l) | 8.5524e-01 (l) | 7.3229e-01 (l) | 1.5814e-01 (l) | 2.7605e-01 (l) | 4.0242e-01 (l) | 1.4101e-01 (l) | 1.3621e-01 (l) | 2.0598e-01 (l) | 4.1926e-01 (l) | 2.5293e-02 (l) | 2.7099e-01 (l) | 5.0000e-02 (l) | 5.0000e-02 (l) | 0.0000e+00 (t) |
13 | 3.4841e-01 (l) | 2.8535e-02 (t) | 1.5419e-01 (l) | 2.5917e-01 (l) | 3.7657e-01 (l) | 1.4952e-01 (l) | 7.4203e-02 (l) | 1.0130e-01 (l) | 8.8528e-02 (l) | 1.6061e-01 (l) | 4.2972e-01 (l) | 1.3876e-01 (l) | 6.1959e-02 (l) | 1.3908e-01 (l) | 3.0000e-02 (l) | 3.0000e-02 (l) | 2.7973e-02 (t) |
14 | 7.7268e-01 (l) | 2.3726e-01 (l) | 1.4813e-01 (t) | 2.7023e-01 (l) | 6.1937e-01 (l) | 2.3112e-01 (l) | 1.0025e-01 (w) | 1.0968e-01 (w) | 1.5941e-01 (l) | 2.8415e-01 (l) | 4.5348e-01 (l) | 2.7379e-01 (l) | 1.6997e-01 (l) | 1.5763e-01 (l) | 1.6000e-01 (l) | 1.5000e-01 (l) | 1.3726e-01 (t) |
15 | 7.9821e+01 (l) | 9.0827e-01 (l) | 1.0463e+00 (l) | 2.1662e+00 (l) | 1.0522e+01 (l) | 1.0616e+00 (l) | 9.0424e-01 (l) | 8.8466e-01 (l) | 7.3805e-01 (t) | 1.0853e+00 (l) | 2.7050e+00 (l) | 8.6952e-01 (t) | 6.4712e-01 (w) | 9.9273e-01 (l) | 4.4000e-01 (w) | 4.2000e-01 (w) | 8.0506e-01 (t) |
16 | 2.7971e+00 (l) | 2.4018e+00 (t) | 2.2131e+00 (w) | 3.4914e+00 (l) | 2.3434e+00 (t) | 1.9837e+00 (w) | 2.3152e+00 (t) | 2.1553e+00 (w) | 1.9729e+00 (w) | 2.8734e+00 (l) | 2.8781e+00 (l) | 1.7489e+00 (w) | 1.9725e+00 (w) | 2.3899e+00 (t) | 2.5200e+00 (t) | 2.5200e+00 (t) | 2.3951e+00 (t) |
17 | 2.7851e+04 (t) | 3.1275e+02 (l) | 1.3597e+01 (w) | 4.0135e+03 (l) | 4.4581e+05 (l) | 2.5879e+02 (l) | 3.8214e+01 (l) | 2.9194e+01 (l) | 1.8521e+01 (t) | 5.2643e+04 (l) | 1.9985e+03 (l) | 2.5739e+02 (l) | 1.2158e+02 (w) | 9.7188e+01 (l) | 3.1229e+06 (l) | 1.5095e+03 (w) | 1.8386e+01 (t) |
18 | 3.1651e+05 (t) | 3.4780e+01 (l) | 7.2311e-01 (w) | 6.8721e+02 (l) | 6.7813e+05 (l) | 2.5187e+01 (l) | 1.7535e+00 (l) | 1.5763e+00 (l) | 8.5047e-01 (t) | 7.4526e+03 (l) | 1.0485e+03 (l) | 3.3189e+01 (l) | 2.8066e+01 (l) | 3.8724e+00 (l) | 1.2933e+04 (l) | 1.0125e+04 (l) | 9.7137e-01 (t) |
19 | 2.6420e+00 (t) | 1.3411e+00 (l) | 3.2565e-01 (w) | 4.0970e+00 (l) | 2.2598e+00 (t) | 1.3184e+00 (l) | 7.7791e-01 (t) | 8.6626e-01 (t) | 4.1739e-01 (w) | 2.1745e+00 (l) | 1.1459e+00 (l) | 2.2578e+00 (l) | 2.2829e-01 (w) | 1.6202e+00 (l) | 5.5000e-01 (w) | 5.5000e-01 (w) | 7.7917e-01 (t) |
20 | 1.1091e+04 (l) | 2.4226e+01 (l) | 2.5344e-01 (w) | 1.5224e+02 (l) | 8.3414e+03 (l) | 1.3382e+01 (l) | 1.2239e+00 (l) | 1.3850e+00 (l) | 4.3605e-01 (w) | 3.0909e+03 (l) | 1.7151e+01 (l) | 1.2599e+01 (l) | 4.1577e+01 (l) | 1.0938e+00 (l) | 9.3693e+03 (l) | 8.8034e+03 (l) | 7.0498e-01 (t) |
21 | 1.7722e+04 (t) | 4.6631e+01 (l) | 1.5300e+00 (l) | 4.2818e+03 (l) | 7.2031e+04 (l) | 9.4695e+01 (l) | 3.7143e+00 (l) | 2.2426e+00 (l) | 2.1863e+00 (l) | 5.2251e+03 (l) | 1.1665e+02 (l) | 1.0997e+02 (l) | 2.0352e+02 (l) | 1.1406e+01 (t) | 1.5952e+07 (l) | 2.0929e+03 (l) | 9.8912e-01 (t) |
22 | 7.9003e+01 (l) | 1.1984e+02 (w) | 4.0742e-01 (w) | 4.2709e+01 (t) | 4.5660e+01 (t) | 3.4109e+01 (w) | 1.6056e+01 (w) | 1.1004e+01 (w) | 2.0681e+00 (t) | 3.7947e+01 (w) | 1.9069e+01 (w) | 3.0770e+01 (w) | 1.3125e+01 (t) | 1.8241e+01 (w) | 1.2790e+02 (w) | 1.2862e+02 (w) | 4.7301e+00 (t) |
23 | 3.3407e+02 (l) | 2.0214e+02 (w) | 3.2946e+02 (w) | 3.2986e+02 (l) | 3.3411e+02 (l) | 3.2946e+02 (l) | 3.2946e+02 (l) | 3.2946e+02 (l) | 3.2946e+02 (t) | 3.2946e+02 (l) | 3.2300e+02 (t) | 3.2956e+02 (l) | 3.2946e+02 (l) | 3.2946e+02 (t) | 2.0000e+02 (w) | 2.0000e+02 (w) | 3.2946e+02 (t) |
24 | 1.3208e+02 (l) | 1.5608e+02 (l) | 1.1504e+02 (l) | 1.4048e+02 (l) | 1.3620e+02 (l) | 1.2757e+02 (l) | 1.1204e+02 (l) | 1.1220e+02 (l) | 1.0923e+02 (l) | 1.3251e+02 (l) | 1.3868e+02 (l) | 1.2406e+02 (l) | 1.0855e+02 (t) | 1.2089e+02 (l) | 1.1744e+02 (l) | 1.1744e+02 (l) | 1.0846e+02 (t) |
25 | 1.7666e+02 (l) | 1.5905e+02 (l) | 1.3970e+02 (t) | 1.7162e+02 (l) | 1.7090e+02 (l) | 1.7970e+02 (l) | 1.6729e+02 (l) | 1.4058e+02 (t) | 1.1613e+02 (w) | 1.8403e+02 (l) | 1.4973e+02 (l) | 1.8632e+02 (l) | 1.7606e+02 (l) | 1.4864e+02 (l) | 1.4781e+02 (l) | 1.4524e+02 (l) | 1.3083e+02 (t) |
26 | 1.0038e+02 (l) | 1.0003e+02 (t) | 1.0015e+02 (l) | 1.0027e+02 (l) | 1.0041e+02 (l) | 1.0017e+02 (l) | 1.0008e+02 (l) | 1.0010e+02 (l) | 1.0009e+02 (l) | 1.0014e+02 (l) | 1.0041e+02 (l) | 1.0013e+02 (l) | 1.0005e+02 (l) | 1.0013e+02 (l) | 1.0005e+02 (l) | 1.0005e+02 (l) | 1.0003e+02 (t) |
27 | 1.4929e+02 (w) | 1.8371e+02 (w) | 1.1107e+02 (w) | 3.5280e+02 (l) | 3.8543e+02 (l) | 3.2132e+02 (l) | 8.9778e+01 (t) | 5.8410e+01 (l) | 1.7852e+01 (t) | 2.8152e+02 (l) | 9.1119e+00 (l) | 2.5600e+02 (l) | 1.8598e+02 (w) | 1.6910e+02 (w) | 2.0000e+02 (w) | 2.0000e+02 (w) | 2.5705e+01 (t) |
28 | 4.3055e+02 (l) | 2.9229e+02 (w) | 3.7206e+02 (l) | 7.3672e+02 (l) | 4.5149e+02 (l) | 3.4854e+02 (l) | 3.8321e+02 (l) | 3.8081e+02 (l) | 3.6154e+02 (l) | 4.8204e+02 (l) | 3.0671e+02 (w) | 4.2341e+02 (l) | 3.8885e+02 (l) | 4.0599e+02 (l) | 2.0000e+02 (w) | 2.0000e+02 (w) | 3.1292e+02 (t) |
29 | 1.2941e+05 (w) | 2.0000e+02 (l) | 2.1815e+02 (l) | 3.1238e+02 (l) | 4.7797e+04 (l) | 2.1175e+02 (l) | 2.2242e+02 (l) | 2.2212e+02 (l) | 1.8295e+02 (t) | 4.1909e+02 (l) | 2.2139e+02 (l) | 3.5544e+05 (l) | 2.3243e+02 (l) | 3.4030e+04 (l) | 2.0000e+02 (l) | 2.0000e+02 (l) | 1.9812e+02 (t) |
30 | 1.6476e+03 (w) | 2.1687e+02 (w) | 4.7501e+02 (l) | 9.4760e+02 (l) | 1.6034e+03 (w) | 3.9582e+02 (l) | 4.6841e+02 (l) | 4.6538e+02 (l) | 5.0008e+02 (l) | 1.7725e+03 (w) | 3.9700e+02 (l) | 6.3914e+02 (l) | 6.2537e+02 (l) | 5.5999e+02 (l) | 2.0000e+02 (w) | 2.0000e+02 (w) | 2.3971e+02 (t) |
w/t/l | 4/4/22 | 5/7/18 | 9/7/14 | 0/2/28 | 1/6/23 | 3/1/26 | 2/3/25 | 3/5/22 | 6/7/17 | 2/0/28 | 3/1/26 | 3/2/25 | 5/7/18 | 3/5/22 | 8/1/21 | 10/1/19 | 0/30/0 |
There are two approaches to tackle such step:
- Calculate the mean directly for all problems by algorithm from the table & returning the algorithm with the least mean
- Calculate the ranks (as in Friedman test) for each problem and then calculate the mean for all problems by algorithm from the table & returning the algorithm with the least ranked mean
The second approach is used by default.
Once the best algorithm is identified, we can now use wilcoxon test, by comparing each algorithm with the best algorithm and identify any significant difference between them. The hypothesis columns denotes whether the p-value poses any significant difference or not
The following table indicate that MVMO has insignificant difference with the best algorithm UMOEAS (that has a p-value of 1), hence, we accept the null hypothesis between MVMO and UMOEAS observation.
W+ denotes the sum of positive ranks, while W- denotes the sum of negative ranks.
The sign of W value can be obtained by subtracting the optimal algorithm from the other algorithm.
Recall that W denotes the rank itself rather than the value of the observation.
It is expected in a true null hypothesis to have similar values between W+ and W-.
Wilcoxon test
Dimension: 10
Parameter: 8
P-Value | Hypothesis | W+ | W- | |
---|---|---|---|---|
b3e3pbest | 1.7344e-06 | (X) | 465 | 0 |
CMLSP | 2.5576e-03 | (X) | 387 | 78 |
DE_b6e6rlwithrestart | 2.9770e-02 | (X) | 348 | 117 |
FCDE | 1.7344e-06 | (X) | 465 | 0 |
FERDE | 1.9209e-06 | (X) | 464 | 1 |
FWA-DM | 5.2165e-06 | (X) | 454 | 11 |
GaAPADE | 1.7988e-05 | (X) | 441 | 24 |
L-SHADE | 2.0603e-05 | (X) | 449 | 16 |
MVMO | 6.7328e-01 | (✓) | 267 | 198 |
NRGA | 1.7344e-06 | (X) | 465 | 0 |
OptBees | 1.0570e-04 | (X) | 421 | 44 |
POBL_ADE | 4.2857e-06 | (X) | 456 | 9 |
rmalschcma | 3.3902e-03 | (X) | 383 | 82 |
RSDE | 6.5213e-06 | (X) | 460 | 5 |
SOO | 1.3820e-03 | (X) | 388 | 77 |
SOO+BOBYQA | 1.5927e-03 | (X) | 386 | 79 |
UMOEAS | 1.0000e+00 | (✓) | 0 | 0 |
Friedman test returns the p-value and statistic between a set of group, effectively identifying if there is any significant difference between all of these groups in general or not.
Unfortunately, the current implementation of Friedman test in Python doesn't implicitly provide any ranking, this has to be done manually. So we can append the Friedman test results to the ranking.
The ranking procedure is thus:
- Applying the Pandas ranking function on each row (problem) based on the cell's mean, ranging from 1 (best), to 17( worst)
- Calculating the mean for all ranks by aggregating all the problems
- The lowest ranked mean is considered as the best algorithm
Notice that UMOEAS is the best algorithm, this is because it has the lowest ranking (lower mean → lower ranking).
The p-value indicates that there is a huge significant difference (converges to zero), this means that there is a wide margin in performance between the algorithms in general.
Friedman test
Dimension: 10
Parameter: 8
Algorithm | |
---|---|
b3e3pbest | 14.97 |
CMLSP | 7.217 |
DE_b6e6rlwithrestart | 5.433 |
FCDE | 14.23 |
FERDE | 15 |
FWA-DM | 9.1 |
GaAPADE | 7.033 |
L-SHADE | 5.867 |
MVMO | 4.667 |
NRGA | 12.7 |
OptBees | 10.7 |
POBL_ADE | 11 |
rmalschcma | 6.017 |
RSDE | 8.767 |
SOO | 9 |
SOO+BOBYQA | 8.2 |
UMOEAS | 3.1 |
P-Value | 4.4440015978556423e-44 (X) |
Statistic | 2.5032e+02 |
The purpose of post hoc tests is to determine exactly which treatment conditions are significantly different, by providing an adjusted p-values.
we will be using the following post-hoc tests
- Bonferroni
- Holm
- Simes-Hochberg
- Hommel
- Nemenyi
- Nemenyi-Friedman
Notice that some algorithms were considered significantly different, but after inspecting the majority of the adjusted p-values, it indicates that it there was no significant difference.
You may note that nemenyi-friedman does not have values less than 10-3, this is due to the psturng
method
nature (embedded within the posthoc_nemenyi_friedman
method), according to the
documentation, the
return value only varies from 0.001 to 0.9, when v>1 (which is under our circumstances equals to np.inf
), refer to the
following docstring:
p : (scalar, array_like) 1. - area from zero to q under the Studentized Range distribution. When v == 1, p is bound between .001 and .1, when v > 1, p is bound between .001 and .9. Values between .5 and .9 are 1st order appoximations.
Post-hoc tests
Dimension: 10
Parameter: 8
unadjusted-p | bonferroni | holm | simes-hochberg | hommel | nemenyi | nemenyi-friedman | verdict | |
---|---|---|---|---|---|---|---|---|
b3e3pbest | 1.7344e-06 (X) | 2.7750e-05 (X) | 2.7750e-05 (X) | 2.4282e-05 (X) | 2.1194e-05 (X) | 6.1459e-05 (X) | 1.0000e-03 (X) | (X) |
CMLSP | 2.5576e-03 (X) | 4.0922e-02 (X) | 1.0231e-02 (X) | 1.0171e-02 (X) | 7.6729e-03 (X) | 1.6005e-01 (✓) | 1.9099e-03 (X) | (X) |
DE_b6e6rlwithrestart | 2.9770e-02 (X) | 4.7633e-01 (✓) | 5.9541e-02 (✓) | 5.9541e-02 (✓) | 5.9541e-02 (✓) | 6.8953e-01 (✓) | 1.0035e-01 (✓) | (✓) |
FCDE | 1.7344e-06 (X) | 2.7750e-05 (X) | 2.7750e-05 (X) | 2.4282e-05 (X) | 2.1194e-05 (X) | 1.8330e-04 (X) | 1.0000e-03 (X) | (X) |
FERDE | 1.9209e-06 (X) | 3.0735e-05 (X) | 2.7750e-05 (X) | 2.4972e-05 (X) | 2.3051e-05 (X) | 9.4760e-05 (X) | 1.0000e-03 (X) | (X) |
FWA-DM | 5.2165e-06 (X) | 8.3464e-05 (X) | 5.7381e-05 (X) | 5.7381e-05 (X) | 5.2165e-05 (X) | 5.4575e-02 (✓) | 1.0000e-03 (X) | (X) |
GaAPADE | 1.7988e-05 (X) | 2.8782e-04 (X) | 1.6190e-04 (X) | 1.6190e-04 (X) | 1.4391e-04 (X) | 2.8040e-01 (✓) | 1.0000e-03 (X) | (X) |
L-SHADE | 2.0603e-05 (X) | 3.2964e-04 (X) | 1.6482e-04 (X) | 1.6482e-04 (X) | 1.6482e-04 (X) | 3.4367e-01 (✓) | 1.0000e-03 (X) | (X) |
MVMO | 6.7328e-01 (✓) | 1.0000e+00 (✓) | 6.7328e-01 (✓) | 6.7328e-01 (✓) | 6.7328e-01 (✓) | 4.2032e-01 (✓) | 2.0124e-01 (✓) | (✓) |
NRGA | 1.7344e-06 (X) | 2.7750e-05 (X) | 2.7750e-05 (X) | 2.4282e-05 (X) | 2.1194e-05 (X) | 3.6588e-04 (X) | 1.0000e-03 (X) | (X) |
OptBees | 1.0570e-04 (X) | 1.6911e-03 (X) | 7.3987e-04 (X) | 7.3987e-04 (X) | 7.3987e-04 (X) | 3.1035e-03 (X) | 1.0000e-03 (X) | (X) |
POBL_ADE | 4.2857e-06 (X) | 6.8571e-05 (X) | 5.1428e-05 (X) | 5.1428e-05 (X) | 4.2857e-05 (X) | 4.5255e-03 (X) | 1.0000e-03 (X) | (X) |
rmalschcma | 3.3902e-03 (X) | 5.4244e-02 (✓) | 1.0231e-02 (X) | 1.0171e-02 (X) | 1.0171e-02 (X) | 2.2516e-01 (✓) | 1.9099e-03 (X) | (X) |
RSDE | 6.5213e-06 (X) | 1.0434e-04 (X) | 6.5213e-05 (X) | 6.5213e-05 (X) | 6.5213e-05 (X) | 1.4512e-01 (✓) | 1.0000e-03 (X) | (X) |
SOO | 1.3820e-03 (X) | 2.2113e-02 (X) | 8.2922e-03 (X) | 7.9635e-03 (X) | 5.6504e-03 (X) | 4.9509e-03 (X) | 1.0000e-03 (X) | (X) |
SOO+BOBYQA | 1.5927e-03 (X) | 2.5483e-02 (X) | 8.2922e-03 (X) | 7.9635e-03 (X) | 6.3708e-03 (X) | 1.2949e-02 (X) | 1.0140e-03 (X) | (X) |
UMOEAS | 1.0000e+00 (✓) | 1.0000e+00 (✓) | 1.0000e+00 (✓) | 1.0000e+00 (✓) | 1.0000e+00 (✓) | 1.0000e+00 (✓) | 1.0000e+00 (✓) | (✓) |
After conducting the tests and followed the above steps, we conclude the following:
-
Some algorithms do not have observations in 100D.
-
The data does not represent the normal distribution, non parametric tests is required.
-
The best algorithm may differ when changing the dimension and/or parameter.
-
The algorithm that has the following characteristics is most likely the best algorithm.
- Relatively the lowest mean.
- Relatively the lowest rank (correlated with the previous point).
- The whisker converges to zero.
- Little to no outliers.
- If any outliers exist, it is not distant from the maximum whisker.
-
Some algorithms perform terribly when the dimension increases.
-
Other algorithms adapt to higher dimensionality and perform surprisingly better (optimized to tackle the 'curse of dimensionality').
-
Some algorithms scored 0/0/30 in the w/t/l comparison, this might change upon modifying the dimension and the parameter.
-
Scoring 0/0/30 does not necessarily mean it's a bad algorithm, perhaps it secured the second place in all problems.
-
Some algorithms came close to the p-value of the best algorithm and found insignificant difference with it, while some had massive significant difference.
-
The p-value of Friedman test always converges to zero, no matter the provided dimension and parameter.
-
Nemenyi-Friedman did not have similar values with the other post-hoc tests, but it was mostly in concurrence with the majority.
-
Nemenyi-Friedman is preferred over Nemenyi when there is a significant difference in the groups (which is true in this case), hence, raw Nemenyi should not be trusted, refer to the following quote from the
scikit_posthocs.posthoc_nemenyi_friedman
documentation.Calculate pairwise comparisons using Nemenyi post hoc test for unreplicated blocked data. This test is usually conducted post hoc if significant results of the Friedman’s test are obtained.
Upon utilizing the created methods, we identified the following algorithms as the best, by plotting the following figures:
![]()
Identify the best algorithm per dimension by observing the algorithm with the highest wins for the respected dimension (color coded).
![]()
This figure represents the same data as the previous plot, but the dataframe is transposed and the bars are stacked, best suited to view each algorithm share as the best algorithm for each dimension.