Separate test definitions from test execution so that providers can use the definitions in their tests

[thomas11]

We'd like to test providers against the examples in this repository as part of their CI process, i.e., before they are released. In order to do that, this PR separates the test suite in misc/test into test definitions and test execution. The execution part then consists of a single test method instead of one per example, which looks like this:

	for _, examples := range definitions.GetTestsForTag("azure") {
		for _, example := range examples {
			run(t, example)
		}
	}

Due to the use of subtests (r.Run()) the results are still cleanly reported and individual tests can be run.

Providers can then "github.com/pulumi/examples/misc/test/definitions" and use a similar snippet to run the same tests, but with their own configuration to use local dependencies and SDKs. See pulumi/pulumi-azure-native#3065 for an example.

There is a tag-based API to retrieve desired test definitions by programming language, provider, or both.

This PR is limited to Azure and Azure Native so far. Other providers can be added as needed.

[mjeffryes]

This all seems reasonable to me assuming docs team agrees.

[cnunciato]

It looks like the PR tests are "passing", but I see this in the logs:

sorry, accidentally edited - thomas11

Is this expected?

[thomas11]

It looks like the PR tests are "passing", but I see this in the logs:

sorry, accidentally edited - thomas11

Is this expected?

No, sorry for the breakage. Fixed now.

[cnunciato]

Looking at the list of tests, it looks like this job (which runs the Azure TS tests) didn't find any tests to run:

https://github.com/pulumi/examples/actions/runs/7919098169/job/21619086583?pr=1583

Run make specific_test_set TestSet=AzureTs
echo "running AzureTs Acceptance Tests"
running AzureTs Acceptance Tests
cd misc/test && go test -json . --timeout 4h -v -count=1 -short -parallel 40 --tags=all --run=TestAccAzureTs | gotestfmt
📦 github.com/pulumi/examples/misc/test
testing: warning: no tests to run
ok  	github.com/pulumi/examples/misc/test	0.031s [no tests to run]

However we do have Azure TS tests, as this PR shows:

https://github.com/pulumi/examples/actions/runs/7626653760/job/20773536499?pr=1565

Run make specific_test_set TestSet=AzureTs
echo "running AzureTs Acceptance Tests"
running AzureTs Acceptance Tests
cd misc/test && go test -json . --timeout 4h -v -count=1 -short -parallel 40 --tags=all --run=TestAccAzureTs | gotestfmt
📦 github.com/pulumi/examples
✅ TestAccAzureTsAppService (7m6.22s)
✅ TestAccAzureTsAppServiceDocker (4m3.78s)
✅ TestAccAzureTsArmTemplate (4m1.79s)
✅ TestAccAzureTsCallAzureSdk (3m41.63s)
✅ TestAccAzureTsFunctions (5m1.89s)
✅ TestAccAzureTsStreamAnalytics (6m53.32s)
✅ TestAccAzureTsVmScaleset (7m56.72s)
✅ TestAccAzureTsWebserver (9m8.56s)

Is there something else that needs to be done to ensure this repo continues to run all of the tests it's currently running?

[thomas11]

Looking at the list of tests, it looks like this job (which runs the Azure TS tests) didn't find any tests to run:
Is there something else that needs to be done to ensure this repo continues to run all of the tests it's currently running?

That was another bug, apologies. Fixed. I've now gone over the PR again, re-organized and cleaned it up a bunch.