This document attempts to summarize Andrew Ng's recommended machine learning workflow from his "Nuts and Bolts of Applying Deep Learning" talk at Deep Learning Summer School 2016. Any errors or misinterpretations are my own.

1. Measure Human-level performance on your task (More)
2. Do your training and test data come from the same distribution?
   • Yes
   • No

The real goal of measuring human-level performance is to estimate the Bayes Error Rate. Knowing your Bayes Error Rate helps you figure out if your model is underfitting or overfitting your training data. More specifically, it will let us measure 'Bias' (as Ng defines it), which we use later in the workflow.

Ng recommends a Train / Dev / Test split of approximately 70% / 15% / 15%.

Calculate your bias and variance as:

• Bias = (Training Set Error) - (Human Error)
• Variance = (Dev Set Error) - (Training Set Error)

An example of high bias:

Fix high bias before going on to the next step.

An example of high variance:

Once you fix your high variance then you're done!

If your train and test data come from different distributions, make sure at least your dev and test sets are from the same distribution. You can do this by taking your test set and using half as dev and half as test.

Carve out a small portion of your training set (call this Train-Dev) and split your Test data into Dev and Test:

|---------------------------------|-----------------------|
|     Train (Distribution 1)      | Test (Distribution 2) |
|---------------------------------|-----------------------|
|  Train              | Train-Dev |  Dev      |    Test   |
|---------------------------------|-----------------------|

Calculate these metrics to help know where to focus your efforts:

An example of high bias:

Fix high bias before going on to the next step.

An example of high variance:

Fix your high variance before going on to the next step.

An example of train/test mismatch:

Fix your train/test mismatch before going on to the next step.

An example of overfitting your dev set:

Once you fix your dev set overfitting, you're done!

Ng suggests these ways for fixing a model with high bias:

• Try a bigger model
• Try training longer
• Try a new model architecture (this can be hard)

Ng suggests these ways for fixing a model with high variance:

• Get more data
  • This includes data synthesis and data augmentation
• Try adding regularization
• Try early stopping
• Try new model architecture (this can be hard)

Ng suggests these ways for fixing a model with high train/test mismatch:

• Try to get more data similar to your test data
• Try data synthesis and data augmentation
• Try new model architecture (this can be hard)

Ng suggests only one way of fixing dev set overfitting:

• Get more dev data

Presumably this would include data synthesis and data augmentation as well.