Export: Introduce Dependency Tree

[NikCharlebois]

Description

When extracting configuration from an existing tenant, we should do the extraction in a logical order to ensure items that have dependencies on others appear first in the configuration so that they are executed sequentially OR we should add DependsOn clause to resources which depends on others. This will get rid of errors when trying to re-apply a freshly exported configuration over another tenant. A concrete example:

Assume you've done a full export of a tenant's EXO workload. The config contains the following blocks in order:
EXOAntiPhishPolicy 9ab4c66f-f6ed-45cc-af52-b14daf12fc0d
{
AdminDisplayName = "Default monitoring policy";
AuthenticationFailAction = "MoveToJmf";
EnableAntispoofEnforcement = $True;
Enabled = $null;
EnableMailboxIntelligence = $null;
EnableOrganizationDomainsProtection = $null;
EnableSimilarDomainsSafetyTips = $null;
EnableSimilarUsersSafetyTips = $null;
EnableTargetedDomainsProtection = $null;
EnableTargetedUserProtection = $null;
EnableUnusualCharactersSafetyTips = $null;
Ensure = "Present";
ExcludedDomains = $null;
ExcludedSenders = $null;
GlobalAdminAccount = $Credsglobaladmin;
Identity = "Monitor Policy";
MakeDefault = $null;
PhishThresholdLevel = "1";
TargetedDomainActionRecipients = $null;
TargetedDomainProtectionAction = "NoAction";
TargetedDomainsToProtect = $null;
TargetedUserActionRecipients = $null;
TargetedUserProtectionAction = "NoAction";
TargetedUsersToProtect = $null;
}
[...]
O365OrgCustomizationSetting 5a6c388e-b42d-43fc-b871-384bcb21ae13
{
Ensure = "Present";
GlobalAdminAccount = $CredsGlobaladmin;
IsSingleInstance = "Yes";
}

Running this command on a new tenant will fail with an error stating that Customization has to be enabled on the tenant first. The customization is enabled by the O365OrgCustomizationSetting resource, which will only be executed after the error occurred. Therefore, we need to either make that resource appear before the EXOAntiphishPolicy one or add a DependsOn clause to it that will be dependent on the O365OrgCustomizationSetting one to ensure a proper execution sequence.

[andikrueger]

I see a chance in extending the settings.json with a new property "resourceDependency". This property could be used within Export-M365DSCConfiguration cmdLet.

{
    "resourceName": "EXOAntiPhishPolicy",
    "description": "",
    "permissions": [
        {
            "read": [],
            "update": []
        }
    ],
   "resourceDependency": [
         "O365OrgCustomizationSetting"
   ]
}

I could think of the following solution approach:

1. Get all resources from the Components parameter
2. Check for dependencies within the settings.json
3. Create an new array to hold all the components and add the dependencies prior to the component position.
4. reduce the array to only hold unique items by deleting all proceeding occurrences of a component.
5. start the export in the sequence of the components within the array.

Maybe we should add a line of comment prior to the resource within the configuration to mention the dependency.

Calling with the maybe newly introduces parameter -ExportResourceDependencies

Export-M365DSCConfiguration -Components @("EXOAntiPhishPolicy") -Credential $Credential -ExportResourceDependencies

would lead to a component list:

if($ExportResourceDependencies)
   $componentsToExport = ["O365OrgCustomizationSetting","EXOAntiPhishPolicy"]
else
   $componentsToExport = ["EXOAntiPhishPolicy"]

and the user would get an information about the new component being added.

The export should look similar to this:

# This resource has dependencies on the following components: 
# - O365OrgCustomizationSetting
# Please make sure these resources are in the right order.

EXOAntiPhishPolicy 'ConfigureAntiphishPolicy'
        {
            Identity                              = "Our Rule"
            MakeDefault                           = $null
            PhishThresholdLevel                   = 1
            EnableTargetedDomainsProtection       = $null
            Enabled                               = $null
            TargetedDomainsToProtect              = $null
            EnableSimilarUsersSafetyTips          = $null
            ExcludedDomains                       = $null
            TargetedDomainActionRecipients        = $null
            EnableMailboxIntelligence             = $null
            EnableSimilarDomainsSafetyTips        = $null
            AdminDisplayName                      = ""
            AuthenticationFailAction              = "MoveToJmf"
            TargetedUserProtectionAction          = "NoAction"
            TargetedUsersToProtect                = $null
            EnableTargetedUserProtection          = $null
            ExcludedSenders                       = $null
            EnableOrganizationDomainsProtection   = $null
            EnableUnusualCharactersSafetyTips     = $null
            TargetedUserActionRecipients          = $null
            Ensure                                = "Present"
            Credential                            = $credsGlobalAdmin
        }

[andikrueger]

The dependency should indicate that this resource depends on another resource. The other resource must be present or executed before this one.

With this in mind I would go for another property naming: requiredResources

Still, I would make the dependency extraction optional.

[ricmestre]

@andikrueger Didn't know this issue was opened but it would be phenomenal if it's implemented.

Currently I'm compiling a list, still very tiny, of what depends on what, extract the dependencies out of the blueprint, on some of them even remove properties (e.g. Members, MemberOf on AADGroup) deploy them and then redeploy all resources again with the missing properties back in if they were previously removed.

And yes I'd make this optional, the example given by @NikCharlebois is on point since they are from 2 different workloads and customers may not want to change/manage all of them through M365DSC, another example is having Intune resources with assignments which depend on AADGroup but they only want to manage Intune resources. Of course in this latter example someone would need to ensure the groups are manually created otherwise the assignments are not done.

[ricmestre]

@andikrueger In this case it would be a mix of all solutions, the json would require a variable to define on which resource(s) another resource might depend on for both add/update and removal since they might be different, but also include what's the corresponding key of the dependency in order to find it.

Note that if the corresponding dependency doesn't exist in the blueprint then the addition of DependsOn would be skipped and we could add a message about it like yours.

Food for thought, currently I'm doing something similar like this internally.

Example, TeamsTenantNetworkSite to be created might have up to 4 different dependencies, if they are all present in the blueprint then DependsOn could be injected as follows.

TeamsTenantNetworkSite "TeamsTenantNetworkSite-TeamsTenantNetworkSite_1"
{
    ApplicationId              = $TeamsApplicationId;
    CertificateThumbprint      = $TeamsCertThumbprint;
    DependsOn                  = @("[TeamsEmergencyCallingPolicy]TeamsEmergencyCallingPolicy-TeamsEmergencyCallingPolicy_1","[TeamsEmergencyCallRoutingPolicy]TeamsEmergencyCallRoutingPolicy-TeamsEmergencyCallRoutingPolicy_1","[TeamsTenantNetworkRegion]TeamsTenantNetworkRegion-TeamsTenantNetworkRegion_1","[TeamsNetworkRoamingPolicy]TeamsNetworkRoamingPolicy-TeamsNetworkRoamingPolicy_1");
    EmergencyCallingPolicy     = "TeamsEmergencyCallingPolicy_1";
    EmergencyCallRoutingPolicy = "TeamsEmergencyCallRoutingPolicy_1";
    NetworkRegionID            = "TeamsTenantNetworkRegion_1";
    NetworkRoamingPolicy       = "TeamsNetworkRoamingPolicy_1";
    EnableLocationBasedRouting = $False;
    Ensure                     = "Present";
    Identity                   = "TeamsTenantNetworkSite_1";
    TenantId                   = $OrganizationName;
}

When removing the same resource then it would be like this, in order to remove the site then the associated subnet(s) must be removed first.

TeamsTenantNetworkSite "TeamsTenantNetworkSite-TeamsTenantNetworkSite_1"
{
    ApplicationId              = $TeamsApplicationId;
    CertificateThumbprint      = $TeamsCertThumbprint;
    DependsOn                  = @("[TeamsTenantNetworkSubnet]TeamsTenantNetworkSubnet-192.168.0.0");
    EmergencyCallingPolicy     = "TeamsEmergencyCallingPolicy_1";
    EmergencyCallRoutingPolicy = "TeamsEmergencyCallRoutingPolicy_1";
    NetworkRegionID            = "TeamsTenantNetworkRegion_1";
    NetworkRoamingPolicy       = "TeamsNetworkRoamingPolicy_1";
    EnableLocationBasedRouting = $True;
    Ensure                     = "Absent";
    Identity                   = "TeamsTenantNetworkSite_1";
    TenantId                   = $OrganizationName;
}

Dependency tree in json would be something similar to this, notice how for removal the properties need to be reversed, we are looking for a "TeamsTenantNetworkSubnet" where the "TeamsTenantNetworkSite"'s "Identity" is equal to the "NetworkSiteID" inside "TeamsTenantNetworkSubnet".

{
    "createUpdate": [
      {
        "resourceName": "TeamsEmergencyCallingPolicy",
        "property": "EmergencyCallingPolicy"
        "dependencyKey": "Identity",
      },
      {
        "resourceName": "TeamsEmergencyCallRoutingPolicy",
        "property": "EmergencyCallRoutingPolicy"
        "dependencykey": "Identity",
      },
      {
        "resourceName": "TeamsTenantNetworkRegion",
        "property": "NetworkRegionID"
        "dependencykey": "Identity",
      },
      {
        "resourceName": "TeamsNetworkRoamingPolicy",
        "property": "NetworkRoamingPolicy"
        "dependencykey": "Identity",
      }
    ],
    "remove": [
      {
        "resourceName": "TeamsTenantNetworkSubnet",
        "property": "Identity",
        "dependencyKey": "NetworkSiteID"
      }
    ]
}

[andikrueger]

This is very valuable input and outlines the complexity of this topic very well.

My initial thought was to only use the dependency information during exports in a way to ask the users if they want to export the additional resources the specified resources within the export command depend on.

For any other scenario I would say it’s a prerequisite on the user’s site to manage dependencies properly.

This would cover way less cases than what you outlined

[ricmestre]

For my use case on the solution I'm developing I need to take care of this in an automated way since the persons using it are not aware of these dependencies, in fact they are not even using blueprints directly I'm converting them from markdown on the fly and need to have some kind of mechanism that injects that DependsOn information into the generated blueprint.

The other mechanism that I still use for a couple of resources, and referred here previously, while I don't move them into DependsOn is to do an initial deployment of the dependencies but without certain properties filled in, for resources where this actually works, and then on a second deployment do it normally with the properties filled in. Example: Group depends on users, deploy users and group without any members and then make a 2nd deployment and deploy the users again (no modifications will occur) and the group but this time with the members filled in.

[subhashvinjamuri]

@ricmestre / @andikrueger / @NikCharlebois - Is this Dependencies solution in place in current M365DSC ? We are also in same situation, where we want to maintain each workload configuration as a separate. But came across a scenario when , say I am applying a teams config using 'TeamsGroupPolicyAssignment' , it needs the Group to apply the policy to. But if the Group mentioned is not existing in the tenant already, neither initial .mof file generation nor 'start-dscconfiguration' doesnt fail. It jsut completes just giving an error message that mentioned group is not present. Due to this, my Configuraiton now contains that group association (with succeeded pipeline / competed PR), but actually that policy assignment to the group is not present in Tenant. THis will lead to inconsistency. Is it possible to

1. either check dependency at .mof file generation level or Apply level & fail the configuration saying Dependencies missing.
2. Or re-arrange the apply configuration such that Dependencies are created first and then proceed with actual resource creation.
   "DependsOn" keywords doesnt seem to be an acceptable property yet as per : https://microsoft365dsc.com/resources/overview/
   PLease Suggest how to proceed.