Skip to content

A tool to evaluate the performance of various machine learning algorithms and preprocessing steps to find a good baseline for a given task.

License

Notifications You must be signed in to change notification settings

marella/evaluate

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A tool to evaluate the performance of various machine learning algorithms and preprocessing steps to find a good baseline for a given task.

Installation

pip install evaluate

Example

import evaluate
from sklearn import datasets

data = datasets.load_iris()
x, y = data.data, data.target

results = evaluate(task='classification', data=(x, y))
results['test_score'].plot.bar()

Documentation

This tool performs common preprocessing steps such as feature scaling, one-hot encoding etc., and runs various ML algorithms such as Random Forests, SVM etc. It then evaluates the performance of each preprocessing step and ML algorithm and provides scores for each. These results can be used to quickly identify preprocessing steps and ML algorithms that perform well to form a good baseline which can be used to develop better models.

evaluate(task,
         data,
         test_data=.2,
         columns=None,
         preprocessors=None,
         estimators=None)
Args
  • task: 'classification' or 'regression'
  • data: Tuple of x, y used for training the model
  • test_data: Tuple of x, y or a number representing the proportion of data to be used for scoring the model
  • columns: Dictionary of lists mapping column types to column names. If not specified numeric and categorical columns are automatically identified
  • preprocessors: List of names of available preprocessors or a custom Preprocessors object
  • estimators: List of names of available estimators or a custom Estimators object
Returns

Dictionary of pandas DataFrames with estimator names as index and preprocessor names as column names with the following keys:

{
    'test_score': ...,
    'train_score': ...,
    'fit_time': ...,
    'score_time': ...,
}
results = evaluate(...)
assert isinstance(results, dict)
scores = results['test_score']
assert isinstance(scores, pandas.DataFrame)
scores.plot.bar()

Preprocessors

Available Preprocessors

Name Column Type Description
n numeric Handle missing data
n:s numeric Standardize features
c categorical Handle missing data and perform one-hot encoding
o ordinal Handle missing data and perform ordinal encoding
t:c text Convert to a matrix of token counts
t:c=2 text Convert to a matrix of token counts including bigrams
t:t text Convert to a matrix of TF-IDF features
t:t=2 text Convert to a matrix of TF-IDF features including bigrams

Multiple preprocessors can be combined into one by separating them with ,:

results = evaluate(..., preprocessors=['n,c,o', 'n:s,c,o'])

Custom Preprocessors

Custom preprocessors can be added as:

from evaluate import evaluate, Preprocessors

preprocessors = Preprocessors()
preprocessors.add('custom_preprocessor', CustomPreprocessor())
results = evaluate(..., preprocessors=preprocessors)

Name of the custom preprocessor must be unique.

Estimators

Available Estimators

Classification Regression
XGBClassifier XGBRegressor
LGBMClassifier LGBMRegressor
RandomForestClassifier RandomForestRegressor
SVC SVR
LogisticRegression LinearRegression
KNeighborsClassifier KNeighborsRegressor
AdaBoostClassifier AdaBoostRegressor
ExtraTreesClassifier ExtraTreesRegressor
GradientBoostingClassifier GradientBoostingRegressor
DecisionTreeClassifier DecisionTreeRegressor
DummyClassifier DummyRegressor

Custom Estimators

Custom estimators can be added as:

from evaluate import evaluate, Estimators

estimators = Estimators(task='classification')
estimators.add('custom_estimator', CustomEstimator())
results = evaluate(..., estimators=estimators)

Name of the custom estimator must be unique.

About

A tool to evaluate the performance of various machine learning algorithms and preprocessing steps to find a good baseline for a given task.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published