Merge pull request #145 from Learnware-LAMDA/doc_image_experiment

[DOC] Add Image Experiment
Learnware-LAMDA · Dec 13, 2023 · 7081a68 · 7081a68
2 parents 8374c3d + 229e9e2
commit 7081a68
Show file tree

Hide file tree

Showing 2 changed files with 16 additions and 0 deletions.
diff --git a/docs/_static/img/image_labeled.png b/docs/_static/img/image_labeled.png
diff --git a/docs/start/exp.rst b/docs/start/exp.rst
@@ -102,6 +102,22 @@ From the figure above, it is evident that when the user's own training data is l
 Image Experiment
 ====================
 
+For the CIFAR-10 dataset, we sampled the training set unevenly by category and constructed unbalanced training datasets for the 50 learnwares that contained only some of the categories. This makes it unlikely that there exists any learnware in the learnware market that can accurately handle all categories of data; only the learnware whose training data is closest to the data distribution of the target task is likely to perform well on the target task. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with a non-zero probability of sampling on only 4 categories, and the sampling ratio is 0.4: 0.4: 0.1: 0.1. Ultimately, the training set for each learnware contains 12,000 samples covering the data of 4 categories in CIFAR-10.
+
+We constructed 50 target tasks using data from the test set of CIFAR-10. Similar to constructing the training set for the learnwares, in order to allow for some variation between tasks, we sampled the test set unevenly. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with non-zero sampling probability on 6 categories, and the sampling ratio is 0.3: 0.3: 0.1: 0.1: 0.1: 0.1. Ultimately, each target task contains 3000 samples covering the data of 6 categories in CIFAR-10.
+
+With this experimental setup, we evaluated the performance of RKME Image using 1 - Accuracy as the loss.
+
+==================== ==================== ==================== ====================
+ Top-1 Reuse Job Selector Reuse Voting Reuse Best in Market 
+==================== ==================== ==================== ====================
+ 0.406 +/- 0.128 0.406 +/- 0.128 0.310 +/- 0.112 0.304 ± 0.046 
+==================== ==================== ==================== ====================
+
+In some specific settings, the user will have a small number of labelled samples. In such settings, learning the weight of selected learnwares on a limited number of labelled samples can result in a better performance than training directly on a limited number of labelled samples.
+
+.. image:: ../_static/img/image_labeled.png
+ :align: center
 
 Get Start Examples
 =========================