Resource symbolic name

[lwang2016]

Resource Symbolic Name

This article proposes a design for adding resource symbolic names to ARM template language in json.

Challenge

Bicep language makes ARM deployment composition simpler and more intuitive as compared to current template language in json, particulary in defining and referencing resource loops.

DSL compiler translates bicep files into an intermediate language in json that ARM template engine needs to understand as well in order to prepare for resource deployment. However this translation runs into challenges caused by template language's limited capability in referencing entities in resource arrays.

For example, a bicep file that creates multiple storage accounts and outputs their blob endpoints:

parameter storageAccountNames array {
    default: ["account1", "account2"]
}

resource[] storageAccounts 'Microsoft.Storage/storageAccounts@2019-04-01': [
    for storageAccountName in storageAccountNames: {
        name: storageAccountName
        location: resourceGroup().location
        sku: { name: "Standard_LRS" }
        kind: "Storage"
        properties: {
            storageProfile: {
                dataDisks: [
                    {
                        diskSizeGB: 1024
                        createOption: "Empty"
                    }
                ]
            }
        }
    }
]

output blobEndpoints array = [
    for storageAccount in storageAccounts: { storageAccount.primaryEndPoints.blob }
]

Notice Bicep outputs the blob endpoints easily because it can reference storage account resources using symbolic name storageAccounts and enumerate through entities by their symbolic names storageAccount as well.

A corresponding template snippet would look like:

"parameters": {
    "storageAccountNames": {
        "type": "array",
        "defaultValue": ["account1", "account2"]
    }
},
"resources": [
    {
        "type": "Microsoft.Storage/storageAccounts",
        "apiVersion": "2019-04-01",
        "name": "[parameters('storageAccountNames')[copyIndex()]]",
        "location": "[resourceGroup().location]",
        "sku": {
            "name": "Standard_LRS"
        },
        "kind": "Storage",
        "properties": {
            "storageProfile": {
                "dataDisks": [
                    {
                        "diskSizeGB": 1024
                        "createOption": "Empty"
                    }
                ]
            }},
        "copy": {
            "name": "storagecopy",
            "count": "[parameters('storageAccountNames').length]"
        }
    }
],
"outputs": {
    "blobEndpoints": {
        "type": array,
        "value": ?
    }
}

Notice outputs.blobEndpoints.value, there's no easy way in template language to reference the array of storage accounts and their primary endpoints. This is because resources property is defined as an array.

Proposal

Change resources property type to object instead of array. This allows asigning symbolic names via "<symbolic name>": "<resource declaration>" pairs. A resource's symbolic name represents its state at runtime.

Extend template engine so that function reference('<symbolic name>') is able to retrieve resource state using its symbolic name.

Above template snippet would then look like:

"parameters": {
    "storageAccountNames": {
        "type": "array",
        "defaultValue": ["account1", "account2"]
    }
},
"resources": {
    "storageAccounts": {
        "type": "Microsoft.Storage/storageAccounts",
        "apiVersion": "2019-04-01",
        "name": "[parameters('storageAccountNames')[copyIndex()]]",
        "location": "[resourceGroup().location]",
        "sku": {
            "name": "Standard_LRS"
        },
        "kind": "Storage",
        "properties": {
            "storageProfile": {
                "dataDisks": [
                    {
                        "diskSizeGB": 1024
                        "createOption": "Empty"
                    }
                ]
            }
        },
        "copy": {
            "name": "storagecopy",
            "count": "[length(parameters('storageAccountNames'))]"
        }
    }
},
"outputs": {
    "copy": [
        "name": "blobEndpoints",
        "count": "[length(reference('storageAccounts'))]",
        "input": "[reference('storageAccounts')[copyIndex()].primaryEndPoints.blob]"
    ]
}

Compatibility with existing template pipeline

Existing template engine pipeline needs to be extended to support both current (v1) and the new template schema (v2).

Template function reference takes resource name or id as argument, it also needs to be updated to understand resource symbolic names. Consider a scenario in which a resource with symbolic name x and name y and another resource with symbolic name y and name x. Calling reference('x') would be ambiguous because 'x' could either refer to the resource name or symbolic name.

To eliminate the ambiguity, function reference takes symbolic name or resource id as argument for template v2. Template engine can easily tell if a template is v1 or v2 by checking the resources JToken type, array for v1 and object for v2.

Similarly, dependsOn function allows providing either resource name or resource id and the behavior is changed to support either symbolic name or resource id in the new schema.

Current template pipeline has restrictions on certain template functions that they must be evaluated before calculating deployment dependencies in order to have a deterministic dependency graph. For example, resource condition must be evaluated to a boolean value to determine if a resource should be included in dependency; loop count must be evaluated to decide number of resources to be created in deployment.

To address the above restrictions, symbolic name in bicep will be compiled into separate template functions, one for template-phase evaluation the other for deployment-phase evaluation. We introduce a new template function "[resource('symbolicName')]" for template-phase evaluation, function properties must have values already specified in template. We keep "[reference('symbolicName')]" to refer to deployment-phase evaluations.

Symbolic Name Compilation Outputs

Scenarios:

Reference resource name

Bicep: symbolicName.name

Json: "[reference('symbolicName', 'Full').name]"

Reference resource property

Bicep: symbolicName.properties.propertyName

Json: "[reference('symbolicName').propertyName]"

Reference array of resources

Bicep: arraySymbolicName.length

Json: "[length(reference('arraySymbolicName'))]"

Reference an array item

Bicep: arraySymbolicName[index].properties.propertyName

Json: "[reference('arraySymbolicName')[copyIndex()].propertyName]"

Reference nested deployment outputs

Bicep: deploymentSymbolicName.outputs.outputName.value

Json: "[reference('deploymentSymbolicName', 'Full').outputs.outputName.value]"

dependsOn resource

Bicep: dependsOn: [symbolicName1, symbolicName2]

Json: "dependsOn": ["symbolicName1", "symbolicName2"]

[majastrz]

The first argument of the reference function is either a resourceName or resourceIdentifier according to https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-resource#reference. If I have a resource with symbolic name x and name y and another resource with symbolic name y and name x and I call reference('x'), what happens?

I wonder if we could leave the reference function as-is and allow expressions to directly reference properties of the symbolic name. For example let's say we have a symbolic name myStorage, could the following expressions be valid?

myStorage.id
myStorage.properties.primaryEndpoints.blob

[majastrz]

Also wonder if we could allow similar expressions for variables and parameters in the template (assuming they're not ambiguous).

[lwang2016]

I think the design is determined by what do we position IL for down the road. My take is customers should no longer need to compose deployment in json directly once bicep reaches parity with current template language. Customers should simply focus on understanding and using bicep while ignoring its compiled outputs.

With this in mind, the proposal is to make bicep translating logic simpler while leveraging as much as possible capabilities provided by current template language. Extending the existing reference function requires less changes on template engine side.

The case that you mentioned is a very good one. I think it's solvable if bicep compiler generates the json as we have control over the validation and translation logic.

As for simplifying how variables and parameters are referenced, totally with you on that, but I think we should embrace symbolic names and achieve consistent syntax in bicep rather than in IL.

Btw, this would be a great topic to discuss for our design meeting.

[lwang2016]

Updated proposal to include discuss feedback.

[majastrz]

Looks good and made me think of more questions 🙂.

We have other places in the templates that accept "resource name or identifier". One example of that is the dependsOn array. Should we make those also use symbolic names when the user opts into the new syntax? And same for any other functions that follow the same pattern? If the answer is yes, then doing the inventory of impacted functions is probably a worthwhile exercise.

We also have a resources array in v1 syntax that allows the user to declare nested resources. Would we leave that as-is in the v2 syntax or would we also make it an object?

[lwang2016]

We have other places in the templates that accept "resource name or identifier". One example of that is the dependsOn array. Should we make those also use symbolic names when the user opts into the new syntax? And same for any other functions that follow the same pattern? If the answer is yes, then doing the inventory of impacted functions is probably a worthwhile exercise.

We also have a resources array in v1 syntax that allows the user to declare nested resources. Would we leave that as-is in the v2 syntax or would we also make it an object?

Yup, I realized the same thing today when prototyping symbolic names, will list all impacted functions when i got a better idea.

Another annoying thing is deployment and template engines are tied with template v1 classes, so defining v2 counterparts means duplicating a heck lot of code simply because of different function signatures, which I am trying to avoid. The approach I am playing with is to change TemplateResource[] Resources property in Template to JProperty Default from Always, and adding a ResourcesV2 side-by-side.

        /// <summary>
        /// Gets or sets the template resources.
        /// </summary>
        [JsonProperty(Required = Required.Default)]
        public TemplateResource[] Resources { get; set; }

        /// <summary>
        /// Gets or sets the template resources v2.
        /// </summary>
        [JsonProperty(Required = Required.Default)]
        public InsensitiveDictionary<TemplateResource> ResourcesV2 { get; set; }

I am debugging in ExpressionEngine to see how to make it understand symbolic names.

[majastrz]

You will probably need to customize the (de)serialization logic to make that work right. By default it will expect the symbolic names in a property called resourcesV2, which we don't want. I've done some really weird custom stuff with Newtonsoft before, so we should probably talk tomorrow.

[lwang2016]

Updated compatibility with existing template pipeline section to cover proposed change to dependsOn function.

Added section for compilation outputs to specify contracts between bicep compilation and template engine.

[anthony-c-martin]

1. How will we handle this from the template schemas side? Will we have to publish a new set of top-level schemas for the new format?
2. I feel like whether we like it or not customers will discover this enhancement, try to use it, and then expect it to be documented. Is there a plan for that?
3. [reference('symbolicName')].name in your example should be [reference('symbolicName').name] (same for a few other examples)?
4. [reference('symbolicName')].properties.propertyName in your example - does that mean you're assuming 'full' by default with this new overload?
5. What is the syntax like to refer to resources nested inside a parent? Will this also change?
6. I see we can detect reliably whether we're a v2 or v1 template and adjust the behavior accordingly, but is there a case for picking a different name for the function, just to avoid confusing users?
7. Will the new usage of the reference() function allow deferred values? (I'm guessing not, just wanted to check!)
8. Will we be blocking bicep users from calling reference() in bicep code? What about cases where there's a genuine need for the reference function (e.g. referring to a resource outside of the template - I think we allow this, right)?
9. In the example for Reference resource property, you're translating symbolicName.property to symbolicName.properties.property? Is that the plan for bicep?

[lwang2016]

Thanks for your comments, Anthony, See response inline:

1. How will we handle this from the template schemas side? Will we have to publish a new set of top-level schemas for the new format?

Yes, we need to publish new schemas for the resources format change.

2. I feel like whether we like it or not customers will discover this enhancement, try to use it, and then expect it to be documented. Is there a plan for that?

Yes, I can see some customers would still hand-craft their json template leveraging the enhancement even if we encourage them to focus on bicep experience. We should definitely have the enhancement well documented, and also have validation logic in template engine to reject invalid usage.

3. [reference('symbolicName')].name in your example should be [reference('symbolicName').name] (same for a few other examples)?

Good catch! Corrected.

4. [reference('symbolicName')].properties.propertyName in your example - does that mean you're assuming 'full' by default with this new overload?

Yes, the goal is to have a consistent compilation output either when reference symbolicName.name or symbolicName.properties.propertyName.

Thinking about 4. and 9. together, I think we should make 'Full' explicit in compilation outputs: "[reference('symbolicName', 'Full').name]" and "[reference('symbolicName', 'Full').properties.propertyName]". This will also be more consistent with current semantic.

5. What is the syntax like to refer to resources nested inside a parent? Will this also change?

Yes, the nested resources array will also be changed to object allowing customers to specify symbolic names.

6. I see we can detect reliably whether we're a v2 or v1 template and adjust the behavior accordingly, but is there a case for picking a different name for the function, just to avoid confusing users?

The question was discussed in prior meetings. A few reasons that we decided to overload reference function:

• We will encourage customers to make bicep the ARM deployment composing experience, steering away from json format itself. So the decision between new vs. overload becomes less significant in most cases. For customers who still prefer directly drafting json for whatever reason, it should be okay as long as we have overloaded behavior well documented.
• The symbolic name allows retrieving either template-stage or deployment-stage resource properties, the latter is semantically identical to reference function. So overloading reference function can leverage what's already implemented in template/deploy engine the most.

7. Will the new usage of the reference() function allow deferred values? (I'm guessing not, just wanted to check!)

It does. If used in resource properties, a corresponding template reference is created; if in template outputs, a template output reference is created. For either case, deployment engine is able to sort out the dependency and populate output values in a same way it does for template v1.

8. Will we be blocking bicep users from calling reference() in bicep code? What about cases where there's a genuine need for the reference function (e.g. referring to a resource outside of the template - I think we allow this, right)?

Yes, we still allow referring to external resource via reference function.

9. In the example for Reference resource property, you're translating symbolicName.property to symbolicName.properties.property? Is that the plan for bicep?

Good catch, it should be symbolicName.properties.propertyName. See response to 4.

[alex-frankel]

Discussed latest discussion on this topic in #257

cc @lwang2016

Lei - one thing I'm not seeing in this issue is updating/overloading the resourceId() function, do we need to add that?

e.g.:

resource stg '...' = { 
  name: 'stg001'
  ...
}
output stgId string = stg.id

Will stg.id compile to:

resourceId('stg')

or the way I would write it today, which is

resourceId('Microsoft.Storage/storageAccounts', 'stg001')

[lwang2016]

stg.id would be compiled to "[reference('stg', 'Full').id]" in current implementation.

e.g.:

resource stg '...' = { 
  name: 'stg001'
  ...
}
output stgId string = stg.id

Will stg.id compile to:

resourceId('stg')

or the way I would write it today, which is

resourceId('Microsoft.Storage/storageAccounts', 'stg001')

[alex-frankel]

I see - I think this is another one that we'd like to special case. In this case, by using the resourceId() function as it's very commonly used in templates today. What do you think?

[lwang2016]

I see - I think this is another one that we'd like to special case. In this case, by using the resourceId() function as it's very commonly used in templates today. What do you think?

I see what you mean. If we do this resourceId('Microsoft.Storage/storageAccounts', 'stg001'), what does stg001 really mean? Does it refer to the whole json object to the symbolic name? or does it mean the name of the resource that this symbolic name corresponds to? It feels like there's some inconsistency here.

[alex-frankel]

yeah, actually it shouldn't have the type as an argument. It should just be a single argument which is the symbolic name:

resourceId('stg001')

Today, resourceId requires two arguments, so we could overload this with an option with just one argument without being ambiguous

[anthony-c-martin]

Deployment service implementation is complete, and is undergoing internal testing before opening up publicly. Will leave this open to track the codegen implementation in Bicep.

[anthony-c-martin]

Things to consider:

• Enabling support in Bicep means that we will break other consumers of templates (ARM-TTK, Template Analyzer, any other custom solutions running post-build analysis), meaning this should be enabled by a config flag and off by default.
• If it's enabled by a config flag, how do we motivate people to use it?