[DOC] Add Image Experiment

Learnware-LAMDA · Dec 10, 2023 · e1764e3 · e1764e3
1 parent 0ff68a1
commit e1764e3
Show file tree

Hide file tree

Showing 2 changed files with 16 additions and 0 deletions.
diff --git a/docs/_static/img/image_labeled.png b/docs/_static/img/image_labeled.png
diff --git a/docs/start/exp.rst b/docs/start/exp.rst
@@ -62,6 +62,22 @@ Text Experiment
 Image Experiment
 ====================
 
+For the CIFAR-10 dataset, we sampled the training set unevenly by category and constructed unbalanced training datasets for the 50 learnwares that contained only some of the categories. This makes it unlikely that there exists any learnware in the learnware market that can accurately handle all categories of data; only the learnware whose training data is closest to the data distribution of the target task is likely to perform well on the target task. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with a non-zero probability of sampling on only 4 categories, with a sampling ratio of 0.4: 0.4: 0.1: 0.1. Ultimately, the training set for each learnware contains 12,000 samples covering the data of 4 categories in CIFAR-10.
+
+We constructed 50 target tasks using data from the test set of CIFAR-10. Similar to constructing the training set for the learnwares, in order to allow for some variation between tasks, we sampled the test set unevenly. Specifically, the probability of each category being sampled obeys a random multinomial distribution, with non-zero sampling probability on 6 categories, with a sampling ratio of 0.3: 0.3: 0.1: 0.1: 0.1: 0.1. Ultimately, each target task contains 3000 samples covering the data of 6 categories in CIFAR-10.
+
+With this experimental setup, we evaluated the performance of RKME Image using 1 - Accuracy as the loss.
+
+==================== ==================== ==================== ====================
+ Top-1 Reuse Job Selector Reuse Voting Reuse Best in Market 
+==================== ==================== ==================== ====================
+ 0.406 +/- 0.128 0.406 +/- 0.128 0.310 +/- 0.112 0.304 ± 0.046 
+==================== ==================== ==================== ====================
+
+In some specific settings, the user will have a small number of labelled samples. In such settings, learning the weight of selected learnwares on a limited number of labelled samples can result in a better performance than training directly on a limited number of labelled samples.
+
+.. image:: ../_static/img/image_labeled.png
+ :align: center
 
 Get Start Examples
 =========================