feat: Adding ephemeral disk support

[PatrickLang]

Reason for Change:
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/ephemeral-os-disks make deployment and runtime faster.

This is still a WIP for review & testing. Will squash and make commit messages conventional soon.

Issue Fixed:
Fixes #1287

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• adds e2e test - will do on 7/25
• tested upgrade from previous version

Notes:

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: PatrickLang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [PatrickLang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[PatrickLang]

/hold

[codecov]

Codecov Report

Merging #1651 into master will decrease coverage by 0.04%.
The diff coverage is 12.5%.

@@            Coverage Diff             @@
##           master    #1651      +/-   ##
==========================================
- Coverage   76.48%   76.44%   -0.05%     
==========================================
  Files         129      129              
  Lines       19033    19004      -29     
==========================================
- Hits        14558    14527      -31     
- Misses       3682     3690       +8     
+ Partials      793      787       -6

[PatrickLang]

This is ready for review. I will squash and update commit message to feat: after that. I also need to do a few adhoc perf tests so I can clarify what the improvement is in deployment time.

[PatrickLang]

Manual tests looking good. Big improvement in 1st launch time of iis

Clean deployment, nothing running

                    "imageReference": {
                        "publisher": "MicrosoftWindowsServer",
                        "offer": "WindowsServer",
                        "sku": "Datacenter-Core-1809-with-Containers-smalldisk",
                        "version": "17763.557.20190604"
                    }

OS disks were all left at 30Gb which is default for that image

2 Standard_D2s_v3 with ephemeral
pull succeeded in 4m 49s
ready in 4m 55s

2 Standard_D2s_v3 without ephemeral
pull succeeded in 6m 55s
ready in 7m 4s

[PatrickLang]

Nice, container local storage improved from ~100 IOPs to ~3300 IOPs

Results at https://github.com/PatrickLang/diskspd-containers

[PatrickLang]

/remove hold

[PatrickLang]

/hold cancel

[jackfrancis]

lgtm, curious if we should merge this with more documentation language

[PatrickLang]

alright, will get more docs
/hold

[PatrickLang]

example warning:

$ ./bin/aks-engine generate /__w/1/s/gopath/src/github.com/Azure/aks-engine/_output/kubernetes-westeurope-63931.json --output-directory /__w/1/s/gopath/src/github.com/Azure/aks-engine/_output/kubernetes-westeurope-63931
time="2019-07-31T19:39:42Z" level=warning msg="Ephemeral disks are enabled for Agent Pool linuxpool1. This feature in AKS-Engine is experimental, and data could be lost in some cases."
time="2019-07-31T19:39:42Z" level=warning msg="Ephemeral disks are enabled for Agent Pool agentwfast. This feature in AKS-Engine is experimental, and data could be lost in some cases."
2019/07/31 19:39:42 
Kubeconfig:/__w/1/s/gopath/src/github.com/Azure/aks-engine/_output/kubernetes-westeurope-63931/kubeconfig/kubeconfig.westeurope.json

[PatrickLang]

Although many operations are faster, the E2E test pass duration impact is still within stddev

	PR#1651	PR#1700
1.12	0:38:47	0:50:55
1.13	0:44:33	0:39:00
1.14	0:37:10	0:44:55
1.15	0:36:57	0:39:51
1.16	0:51:19	0:49:36
mean	0:41:45	0:44:51
stddev	0:06:10	0:05:27

This PR leaves ephemeral on for the Windows test passes so we can measure if it breaks. If it's too brittle, then we can disable it.

[PatrickLang]

/remove hold

[jackfrancis]

/lgsm

[jackfrancis]

and

[jackfrancis]

/lgtm

[palma21]

@PatrickLang do we have a sample model or doc for this? Wanted to take it for a spin