Learnware
is a model sharing platform, which give a basic implementation of the learnware paradigm. A learnware is a well-performed trained machine learning model with a specification that enables it to be adequately identified to reuse according to the requirement of future users who may know nothing about the learnware in advance. The learnware paradigm can solve entangled problems in the current machine learning paradigm, like continual learning and catastrophic forgetting. It also reduces resources for training a well-performed model.
Machine learning, especially the prevailing big model paradigm, has achieved great success in natural language processing and computer vision applications. However, it still faces challenges such as the requirement of a large amount of labeled training data, difficulty in adapting to changing environments, and catastrophic forgetting when refining trained models incrementally. These big models, while useful in their targeted tasks, often fail to address the above issues and struggle to generalize beyond their specific purposes.
The learnware paradigm introduces the concept of a well-performed, trained machine learning model with a specification that allows future users, who have no prior knowledge of the learnware, to reuse it based on their requirements.
Developers or owners of trained machine learning models can submit their models to a learnware market. If accepted, the market assigns a specification to the model and accommodates it. The learnware market could host thousands or millions of well-performed models from different developers, for various tasks, using diverse data, and optimizing different objectives.
Instead of building a model from scratch, users can submit their requirements to the learnware market, which then identifies and deploys helpful learnware(s) based on the specifications. Users can apply the learnware directly, adapt it using their data, or exploit it in other ways to improve their model. This process is more efficient and less expensive than building a model from scratch.
Benefit | Description |
---|---|
Lack of training data | Strong models can be built with small data by adapting well-performed learnwares. |
Lack of training skills | Ordinary users can obtain strong models by leveraging well-performed learnwares instead of building models from scratch. |
Catastrophic forgetting | Accepted learnwares are always stored in the learnware market, retaining old knowledge. |
Continual learning | The learnware market continually enriches its knowledge with constant submissions of well-performed learnwares. |
Data privacy/ proprietary | Developers only submit models, not data, preserving data privacy/proprietary. |
Unplanned tasks | Open to all legal developers, the learnware market can accommodate helpful learnwares for various tasks. |
Carbon emission | Assembling small models may offer good-enough performance, reducing interest in training large models and the carbon footprint. |
Learnware is currently hosted on PyPI. You can easily intsall Learnware
according to the following steps:
-
For Windows and Linux users:
pip install learnware
-
For macOS users:
conda install -c pytorch faiss pip install learnware
The Learnware Market consists of a wide range of learnwares. A valid learnware is a zipfile which is composed of the following four parts.
-
__init__.py
A python file offering interfaces for your model's fitting, predicting and fine-tuning.
-
rkme.json
A json file containing the statistical specification of your data.
-
learnware.yaml
A config file describing your model class name, type of statistical specification(e.g. Reduced Kernel Mean Embedding,
RKMETableSpecification
), and the file name of your statistical specification file. -
environment.yaml
A Conda environment configuration file for running the model (if the model environment is incompatible, you can rely on this for manual configuration). You can generate this file according to the following steps:
-
Create env config for conda:
conda env export | grep -v "^prefix: " > environment.yaml
-
Recover env from config:
conda env create -f environment.yaml
-
We also demonstrate the detail format of learnware zipfile in [DOC link], and also please refer to Examples for concrete learnware zipfile example.
Users can start an Learnware
workflow according to the following steps:
The EasyMarket
class implements the most basic set of functions in a Learnware
.
You can use the following code snippet to initialize a basic Learnware
named "demo":
import learnware
from learnware.market import EasyMarket
learnware.init()
easy_market = EasyMarket(market_id="demo", rebuild=True)
Before uploading your learnware into the Learnware
,
create a semantic specification semantic_spec
by selecting or filling in values for the predefined semantic tags
to describe the features of your task and model.
For example, the following code snippet demonstrates the semantic specification of a Scikit-Learn type model, which is designed for business scenario and performs classification on tabular data:
semantic_spec = {
"Data": {"Values": ["Tabular"], "Type": "Class"},
"Task": {"Values": ["Classification"], "Type": "Class"},
"Library": {"Values": ["Scikit-learn"], "Type": "Class"},
"Scenario": {"Values": ["Business"], "Type": "Tag"},
"Description": {"Values": "", "Type": "String"},
"Name": {"Values": "demo_learnware", "Type": "String"},
}
Once the semantic specification is defined, you can easily upload your learnware with a single line of code:
easy_market.add_learnware(zip_path, semantic_spec)
Here, zip_path
is the directory of your learnware zipfile.
To search for learnwares that fit your task purpose,
you should also provide a semantic specification user_semantic
that describes the characteristics of your task.
The Learnware
will perform a first-stage search based on user_semantic
,
identifying potentially helpful leranwares whose models solve tasks similar to your requirements.
# construct user_info which includes semantic specification for searching learnware
user_info = BaseUserInfo(id="user", semantic_spec=semantic_spec)
# search_learnware performs semantic specification search if user_info doesn't include a statistical specification
_, single_learnware_list, _ = easy_market.search_learnware(user_info)
# single_learnware_list is the learnware list by semantic specification searching
print(single_learnware_list)
If you choose to porvide your own statistical specification file stat.json
,
the Learnware
can perform a more accurate leanware selection from
the learnwares returned by the previous step. This second-stage search is based on statistical information
and returns one or more learnwares that are most likely to be helpful for your task.
For example, the following code is designed to work with Reduced Set Kernel Embedding as a statistical specification:
import learnware.specification as specification
user_spec = specification.RKMETableSpecification()
user_spec.load(os.path.join(unzip_path, "rkme.json"))
user_info = BaseUserInfo(
semantic_spec=user_semantic, stat_info={"RKMETableSpecification": user_spec}
)
(sorted_score_list, single_learnware_list,
mixture_score, mixture_learnware_list) = easy_market.search_learnware(user_info)
# sorted_score_list is the learnware scores based on MMD distances, sorted in descending order
print(sorted_score_list)
# single_learnware_list is the learnwares sorted in descending order based on their scores
print(single_learnware_list)
# mixture_learnware_list is the learnwares whose mixture is helpful for your task
print(mixture_learnware_list)
# mixture_score is the score of the mixture of learnwares
print(mixture_score)
Based on the returned list of learnwares mixture_learnware_list
in the previous step,
you can easily reuse them to make predictions your own data, instead of training a model from scratch.
We provide two baseline methods for reusing a given list of learnwares, namely JobSelectorReuser
and AveragingReuser
.
Simply replace test_x
in the code snippet below with your own testing data and start reusing learnwares!
# using jobselector reuser to reuse the searched learnwares to make prediction
reuse_job_selector = JobSelectorReuser(learnware_list=mixture_learnware_list)
job_selector_predict_y = reuse_job_selector.predict(user_data=test_x)
# using averaging ensemble reuser to reuse the searched learnwares to make prediction
reuse_ensemble = AveragingReuser(learnware_list=mixture_learnware_list)
ensemble_predict_y = reuse_ensemble.predict(user_data=test_x)
Learnware
also provides an auto workflow example, which includes preparing learnwares, upload and delete learnware from markets, search learnware with semantic specifications and statistical specifications. The users can run examples/workflow_by_code.py
to try the basic workflow of Learnware
.
For all experiments, we used a single linux server. Details on the specifications are listed in the table below. All processors were used for training and evaluating.
System | GPU | CPU |
---|---|---|
Ubuntu 20.04.4 LTS | Nvidia Tesla V100S | Intel(R) Xeon(R) Gold 6240R |
We designed experiments on three publicly available datasets, namely Prediction Future Sales (PFS), M5 Forecasting (M5) and CIFAR 10. For the two sales forecasting data sets of PFS and M5, we divide the user data according to different stores, and train the Ridge model and LightGBM model on the corresponding data respectively. For the CIFAR10 image classification task, we first randomly pick 6 to 10 categories, and randomly select 800 to 2000 samples from each category from the categories corresponding to the training set, constituting a total of 50 different uploaders. For test users, we first randomly pick 3 to 6 categories, and randomly select 150 to 350 samples from each category from the corresponding categories from the test set, constituting a total of 20 different users.
We tested the efficiency of the specification generation and the accuracy of the search and reuse model respectively. The evaluation index on PFS and M5 data is RMSE, and the evaluation index on CIFAR10 classification task is classification accuracy
The time-consuming specification generation is shown in the table below:
Dataset | Data Dimensions | Specification Generation Time (s) |
---|---|---|
PFS | 8714274*31 | < 1.5 |
M5 | 46027957*82 | 9~15 |
CIFAR 10 | 9000*3*32*32 | 7~10 |
The accuracy of search and reuse is shown in the table below:
Dataset | Top-1 Performance | Job Selector Reuse | Average Ensemble Reuse |
---|---|---|---|
PFS | 1.955 +/- 2.866 | 2.175 +/- 2.847 | 1.950 +/- 2.888 |
M5 | 2.066 +/- 0.424 | 2.116 +/- 0.472 | 2.512 +/- 0.573 |
CIFAR 10 | 0.619 +/- 0.138 | 0.585 +/- 0.056 | .715 +/- 0.075 |
We appreciate all contributions and thank all the contributors!
TODO: Here paste the github API after publishing:
Visit LAMDA's official website.