aws-certificatemanager: Support construct to create Certificate from any region, similar to EdgeFunction

[Xarno]

Describe the feature

As requested and liked many times please have the Certificate Construct accept a region setting again, as it has had in DnsValidatedCertificate.
#9274 (comment)

Use Case

Create Cloudfront distributions in one region, create the corresponding certificate in us-east-1.
Create Cognito Userpools with a custom certificate as mostafafarzaneh mentioned in the linked issue.

Proposed Solution

No response

Other Information

Also the current documentation is wrong.
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager.Certificate.html#validation mentions a „region“ property which does not exist in https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager.CertificateValidation.html or https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager.CertificationValidationProps.html

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.76.0

Environment details (OS name and version, etc.)

Mac OS 13.3

[peterwoodworth]

This is how we recommend doing it now https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager-readme.html#cross-region-certificates. If this doesn't work then let us know

[alejandro-gstack]

@peterwoodworth, The way that AWS is proposing managing the certificate in a different stack is not providing a good developer experience because instead of defining in the same CDK stack, we will need to create two stacks, one for creating the ACM certificate in the us-east-1 region that requests for example Cloudfront and then reference it in the other stack built in the desired region.

[peterwoodworth]

I'm curious what the negative experience is exactly with needing to declare another stack in your app, is there a use case where this isn't trivial?

[alejandro-gstack]

Hi @peterwoodworth, consider that you are creating templates (constructs in CDK) to reduce the complexity of deploying infrastructure like a static website (s3/cloudfront/route53/certificate). With the previous approach, developers will create a CDK stack and declare the template and the resources needed in one stack.

Now, they will need to manage two stacks and one stack that only will contain a certificate, and they will need to deploy them in a specific order.

Do you know what would be the suitable infra definition to encapsulate the creation of the certificate in (us-east-1) and all the other resources in a different region? Can I attach the two CDK stacks and programmatically define the order in which they will be deployed?

Checking the documentation, we will need to deploy the CDK stacks separately and in a particular order.

import { aws_cloudfront as cloudfront, aws_cloudfront_origins as origins } from 'aws-cdk-lib';
declare const app: App;

const stack1 = new Stack(app, 'Stack1', {
  env: {
    region: 'us-east-1',
  },
  crossRegionReferences: true,
});
const cert = new acm.Certificate(stack1, 'Cert', {
  domainName: '*.example.com',
  validation: acm.CertificateValidation.fromDns(PublicHostedZone.fromHostedZoneId(stack1, 'Zone', 'ZONE_ID')),
});

const stack2 = new Stack(app, 'Stack2', {
  env: {
    region: 'us-east-2',
  },
  crossRegionReferences: true,
});

new cloudfront.Distribution(stack2, 'Distribution', {
  defaultBehavior: {
    origin: new origins.HttpOrigin('example.com'),
  },
  domainNames: ['dev.example.com'],
  certificate: cert,
});

[alejandro-gstack]

This is a great example of the use case I'm mentioning; what will be the right way to simplify this construct definition using multi-stack deployments?

By the way, I'm not familiar with multi-stack deployments.

https://github.com/aws-samples/aws-cdk-examples/blob/master/typescript/static-site/static-site.ts#L24

[peterwoodworth]

All you should need to do is create a separate stack for your Certificate that deploys into us-east-1, I don't think this would add much complexity in any use case I can imagine - However you define this is up to you. If you're unfamiliar with creating apps with multiple stacks, please see our docs on creating an app with multiple stacks, and having stacks depend on another stack

[peterwoodworth]

For those coming to this thread, please see this comment left by Cory explaining why DnsValidatedCertificate was deprecated. We still stand by this decision. However, as Cory notes in this comment, we would be on board with supporting a construct that handles the creation of the stack in us-east-1 for you. Relabelling to p1, as this is clearly a desired feature

[blimmer]

@peterwoodworth , I think the idea of a construct that assists with the "create a stack in a different region" workaround is a good compromise! That will make handling the deprecation warning much simpler.

[ckifer]

Another use-case for this where creating another stack isn't trivial is when managing the HostedZone in the same stack Cloudfront is in. I can't just create a us-east-1 stack with a Certificate because my HostedZone doesn't exist until it is created in the other-region stack - and I need its information for the cert validation.

I would need to create 3 stacks dependent on one another: HZ creation (in say us-west-2) -> Cert creation (in us-east-1) -> Cloudfront distro (back to us-west-2). Unless I'm missing something and there is an easier way to do that.

edit: I guess moving the HZ setup to the us-east-1 stack solves this. But still

[trautonen]

I've created a construct that handles this with a custom resource: https://github.com/trautonen/cdk-dns-validated-certificate
The construct handles the two most common cases:

• cross account hosted zone based dns validation
• cross region certificates (for example for CloudFront)

It's still lacking alias support, but I'm open to contributions and might have time to add it soon.

[guentherwieser]

Another use case that previously worked with DnsValidatedCertificate, and no longer works when you use multi-stack deployments, is this one here: #26714

We build the certificate's hostname from values from our Parameter Store in eu-central-1, but this fails with multi-stack-deployments with the error "Cross stack references are only supported for stacks deployed to the same account or between nested stacks and their parent stack".

[fdx-sachit]

CDK: "aws-cdk": "2.95.0",
Language: Typescript

I have a static site deployed in Sydney region. I have got a s3 bucket and cloudfront as a pattern. I am also trying to assign it a ssl certificate using ACM. I have a hosted zone in the global region. Now given my app is in Sydney region, and ACM only in us-east-1, i am trying to use the suggested way of doing this.

( I followed this tutorial https://www.subaud.io/blog/deploy-a-cross-region-spa-with-cloudfront )

I am trying the above suggestion on multi region stack. So here is what i've done:

• Initialise a cdk app

const app = new App({
    context: {
        [PERMISSIONS_BOUNDARY_CONTEXT_KEY]: {
            name: PERMISSION_BOUNDARY_NAME,
        },
    },
});

Please note the permission boundary that will become the problem i am having.

• Then i initialise a stack in us-east-1 for the certificate

const certificateStack = new CertificateStack(app, "CertificateStack", {
    env: {
        account: accountId,
        region: 'us-east-1',
    },
    synthesizer: new DefaultStackSynthesizer({
        qualifier: "...",
    }),
    appDomain: ...,
    primaryDomain: ...,
    crossRegionReferences: true,
});

• I have a second stack for my static site deployment as followes

const staticSite = new InfraCoreStack(app, "InfraCoreStack", {
    env: {
        account: accountId,
        region: app.node.tryGetContext("accountId") ?? "ap-southeast-2",
    },
    synthesizer: new DefaultStackSynthesizer({
        qualifier: "....",
    }),
    certificate: certificateStack.siteCertificate,
    crossRegionReferences: true,
});

Now when i synthesise this, it does that without error. But the problem is, because cdk is bootstrapped using a permission boundary, the certificate stack is not attaching that permission boundary to the IAM Role within the certificate stack. And when i try to deploy it, it fails to create the stack with permission errors.

API: iam:CreateRole User: arn:aws:sts::xxxx:assumed-role/xx-xxx-xxxx-xxxxx-us-east-1/AWSCloudFormation is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::xxxx:role/CertificateStack-CustomCrossRegionExportWriterCust-xxxx because no permissions boundary allows the iam:CreateRole action

Has anyone encountered this ? If so, can someone point me in the direction I need to fix this ? It will be a shame if i can't do it this way. Defeats the purpose of me trying to automate this process when I might have to have a certificate manually created and referenced as a parameter.

Thanks

[blimmer]

@peterwoodworth , not sure how much collaboration CDK has with the CFN team within Amazon, but an alternative to working around this in CDK would be to push forward on this CFN request: aws-cloudformation/cloudformation-coverage-roadmap#523

[pergardebrink]

I'm curious what the negative experience is exactly with needing to declare another stack in your app, is there a use case where this isn't trivial?

@peterwoodworth We hit a scenario where this became a real mess.

Given the example stack you referenced earlier here: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager-readme.html#cross-region-certificates

If I deploy that stack with a hostname (test.example.com) for my CloudFront distribution and Certificate, and then at a later stage, I need to rename my hostname (to test2.example.com) so I just update the properties.

Then when I try to deploy, it will fail with the following message:

0/4 Currently in progress: Stack1, Cert5C9FAEC1
Stack1 | 1/4 | 03:00:08 | UPDATE_COMPLETE      | AWS::CertificateManager::Certificate | Cert (Cert5C9FAEC1)
Stack1 | 1/4 | 03:00:10 | UPDATE_IN_PROGRESS   | Custom::CrossRegionExportWriter      | ExportsWritereuwest142AF533A/Resource/Default (ExportsWritereuwest142AF533A3E3B99E4)
Stack1 | 1/4 | 03:00:18 | UPDATE_FAILED        | Custom::CrossRegionExportWriter      | ExportsWritereuwest142AF533A/Resource/Default (ExportsWritereuwest142AF533A3E3B99E4) Received response status [FAILED] from custom resource. Message returned: Error: Some exports have changed!
/cdk/exports/Stack2/Stack1useast1RefCert5C9FAEC1924273F6
    at handler (/var/task/index.js:1:683)
    at Runtime.handler (/var/task/__entrypoint__.js:1:938)

The error message from here:

aws-cdk/packages/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/cross-region-ssm-writer-handler/index.ts

Line 26 in 06bea31

throw new Error('Some exports have changed!\n'+ changedExports.join('\n'));

indicates by the comment above that line that the export can never change, which would be a problem for certificates in the cases where you need to modify it (rename, add alternate names, etc).

[dickhardt]

I read Cory's explanation for deprecating DnsValidatedCertificate. I also notice he is no longer at AWS. His rationale certainly does not follow Amazon principals of Invent and Simplify, or Customer Obsession.

Requiring the certificate to be in US-East-1 is not my problem as a customer. That is an AWS limitation that I as a customer should not be having to deal with. That Cloud Formation is going a different way is also not my problem.

Please invent and simplify.

I spent a couple hours I won't get back looking to get rid of the 8

[WARNING] aws-cdk-lib.aws_certificatemanager.DnsValidatedCertificate is deprecated.
  use {@link Certificate } instead
  This API will be removed in the next major release.

messages I see whenever I update my infrastructure.

I don't have any cross account stacks, and I don't want to create any cross account stacks. I want each stack to be in its own region and account for isolation.

In other words, this issue is larger than DnsValidatedCertificate -- it is requiring resources to be in a magic region.

FWIW I was a Security PE in AWS for a while

[aaroncowie]

I'm curious what the negative experience is exactly with needing to declare another stack in your app, is there a use case where this isn't trivial?

All you should need to do is create a separate stack for your Certificate that deploys into us-east-1, I don't think this would add much complexity in any use case I can imagine - However you define this is up to you. If you're unfamiliar with creating apps with multiple stacks, please see our docs on creating an app with multiple stacks, and having stacks depend on another stack

@peterwoodworth I have been attempting to use cross region cross stack references as suggested and this has created significant operational overhead for our deployments. What used to be a largely foolproof automated deployment is now extremely error prone and requires much manual intervention.

All of our resources are in ap-southeast-2 except edge lambdas and cloudfront certs. The edge lambdas work fine since they are implemented transparently from the regional stack, however it is now almost impossible to progress changes through CI that involve changes to cloudfront served domains.

As an example trying to delete a cloudfront distribution is now impossible through a regular CI deploy.

The stack dependency order is correctly expressed for creation, however for deletion of the distribution the regional stack needs to run before the global one otherwise the custom resource created for cross region cross stack references fails.

I discovered that the --exclusively flag can be used to manually resolve this on a case by case basis however this doesn't help for CI which can receive any combination of cdk changes and cannot know the correct order (if any) to execute the stacks. This is also made worse by the fact that anything touching cloudfront is extremely slow to deploy and the custom lambda throws on rollback.

I will probably create a separate issue for the error on rollback of the custom resource once I better understand how it is implemented.

Note that other companies I have spoken to are working around this by using hardcoded arn's and manually deployed resources for anything that is forced to exist in us-east-1. This is hardly the way cdk is intended to work.

[gshpychka]

All you should need to do is create a separate stack for your Certificate that deploys into us-east-1, I don't think this would add much complexity in any use case I can imagine

One use-case is stacksets - you would have to create a separate stackset just for the stack with the Certificate, since stacks deployed by stacksets cannot be environment-specific.

[krailler]

Hello everyone,

As a result of this problem and seeing that CDK should really make things easier instead of making them a little more difficult, I have tried to implement something related to Certificates based on the EdgeFunction of CloudFront, which I think is brilliant, since it transparently generates the certificate in the region where it belongs (which in the past did the DNSValidatedCertificate, but in a different way), without having to do tricks like activating crossRegionReferences.

I want to share the resource to obtain opinions, and see if in the future it would be possible to obtain this same thing inside the cdk lib.

https://gist.github.com/krailler/6cdf11e1a0d64c8f27b29bf82811c513

[benmccallum]

I'm following the suggested approach, multi-stack with crossRegionReferences enabled (C# CDK v2.126.0), but the CrossRegionExportWriter fails to deploy 😮‍💨

Not sure whether to laugh or cry haha. Painful enough to have to deal with the fundamental limitation (which, fair enough, is probably there for reason but should be hidden from us as others have said). So far I've had to:

1. Rip out cert management from where it was nicely co-located with the CF distro into a US East 1 stack.
2. As a result of that, rip out Hosted Zoned management into a precursor stack as well.
3. Have to deal with multiple regions and enable cross-region refs, leading me here.

Created stack arn:aws:cloudformation:us-east-1:<account-id-1>:stack/GlobalPlatformStackUSEast1/<guid> in region us-east-1 
February 16th 2024 14:03:25Info
Current stack state: CustomCrossRegionExportWriterCustomResourceProviderRoleC951B1E1 - AWS::IAM::Role - CREATE_IN_PROGRESS - Resource creation Initiated 
February 16th 2024 14:03:42Info
Current stack state: CustomCrossRegionExportWriterCustomResourceProviderRoleC951B1E1 - AWS::IAM::Role - CREATE_COMPLETE 
February 16th 2024 14:03:47Info
Current stack state: CustomCrossRegionExportWriterCustomResourceProviderHandlerD8786E8A - AWS::Lambda::Function - CREATE_FAILED - Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400, Request ID: b270b70e-40e3-4c31-b82f-10534bedac83)" (RequestToken: 965d38db-8c55-ee6b-d090-75e1c5397f6b, HandlerErrorCode: InvalidRequest) 

Edit: Looks like the Lambda resource references an S3Bucket and S3Key that points to things in an ....assets.json file. I assume my method of CD (Octopus Deploy CloudFormation template) isn't deploying assets, so will figure that out. My bad, but will leave my comment in case anyone ends up in the

Edit2: This means I'd have to use cdk deploy my-stack to deploy assets, rather than just using the built-in step template for deploying CloudFormation templates. I guess I'll have to do a dodgy initial deploy from cmdline and then things will work (until they don't) just deploying the template. Not sure I want to go down to running cdk deploy myself, as I lose a lot of the nice features of the Octo step.

Edit3: It seems this US-East-1 requirement also extends to WAFv2 :/

[chriswilty]

@fdx-sachit #25343 (comment)

I also have this problem: there seems to be no way to get hold of the CrossRegionExportWriter role, in order to attach a permissions boundary.

All I ask is that before the next major release comes along and removes DnsValidatedCertificate, a way is found to allow stack crossRegionReferences to work with Permissions Boundaries, because currently that's a showstopper for us.

[quantfreedom]

be sure to upvote this or thumbs up it or mark it as an answer so others know it works ... only if it works for you of course ... this way we can save others weeks of trying to figure this out

I have figured out a solution and i posted it here in this issue #9274 (comment)

[jalvinake]

If you are planning on deprecating DnsValidatedCertificate, please provide an alternative that we can use in our existing stacks. Moving our cert creating to a separate stack in us-east-1 and then doing cross-stack references is not a practical change. I wasted ~4 hours trying to make this work for an production web application today and have given up. Please reach out if you need more details.

[jcoppieters]

If you are planning on deprecating DnsValidatedCertificate, please provide an alternative that we can use in our existing stacks. Moving our cert creating to a separate stack in us-east-1 and then doing cross-stack references is not a practical change. I wasted ~4 hours trying to make this work for an production web application today and have given up. Please reach out if you need more details.

I couldn't agree more !

[vincent-dm]

I couldn't agree more !

Yes, me (and my colleagues) too! The current deprecation and dubious proposed migration path really throws AWS customers under the bus.

[blimmer]

I still think the best case scenario is that we get native region support in CloudFormation (see aws-cloudformation/cloudformation-coverage-roadmap#523 / #25343 (comment)). With that, this entire problem would go away.

If native support isn't possible before DnsValidatedCertificate is set to be removed in CDK v3, I think we should keep it. For my clients, at least, this issue could prevent us from upgrading.

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[olpeh]

We are also facing this issue but are unable to use the suggested 2 stack solution due to this:

>Error: Stack "stack2" cannot reference {stack1/Certificate/Resource[Ref]} in stack "stack1". Cross stack references are only supported for stacks deployed to the same account or between nested stacks and their parent stack

So, for us, this does not work:
#25343 (comment)

EDIT: never mind, we had forgotten to pass the account to the other stack and thus the error message was valid.

[metametadata]

There's a way to suppress annoying deprecation logs in Clojure/Java by filtering System/err: #24512 (comment).

[nickhudkins]

Could we please get an update on this issue?

[Gribbs]

this is madness.

[ruslan-roboto]

The necessity of having a dedicated cert stack and then using cross-region references has caused us a non-trivial amount of operational burden, and has been a time sink for developers. I understand that there is a lot on your plate and prioritization is more of an art than a science, but any update would be extremely helpful. Thanks!

[wparad]

It for sure is nonsensical for the suggestion to be create two stacks. Multiple stacks is almost always the wrong answer, it's totally unnecessary. Adds significant complexity and creates non-obvious problems. So we know that advice doesn't make sense. However, what is also a problem is that it isn't clear that there is a valid use case to continue to need create a us-east-1 certificate from another region.

The core of this problem comes from the fact that some resources can be deployed in any region through the AWS API, and other resources not. IAM and CloudFront come to mind. When you deploy these resources to AWS it doesn't matter which region you pick. However other resources that interact with them do require a region, it was always the case that certificates had to be in us-east-1 and lambda@edge functions have to be in us-east-1 and I believe there are others as well (I don't want to swear to it but things like WAF Global might also be the case).

I learned a long time ago two things here are 100% wrong:

• The AWS approach defined by the CloudFront API
• And, Because of AWS' approach here, deploying a CloudFront to a non-us-east-1 region

The correct solution is always deploying CloudFront in a us-east-1 region stack.

I believe the real issue is that some people have chosen to deploy CloudFront to a non-us-east-1 region stack, and now still need the cert in us-east-1. Before they had a solution, and now they also have a solution, but they don't like it.

There are 3 different solutions here:

• Recreate a new custom resource to deploy the us-east-1 certificate
• Create multiple stacks
• Migrate your resources to the us-east-1 region

As I pointed out (2) is wrong, and as @dickhardt pointed out (1) is also wrong. That leaves us with (3) the least wrong solution.

Because of the implementation AWS CloudFront is telling us to always use the us-east-1 region for our CloudFront stacks. I know that it is dumb and we don't want to do that, but asking the CDK to magically come to the rescue isn't the core problem. The CDK should help us make CloudFormation more bearable when it can, but the built-in custom resource was definitely a hack. Adding it back in is adding another hack ontop of the real problem. The reason the hack ever existed was to fix a completely different problem. The fact this supported the non-us-east-1 seems like merely a coincidence. So, in order for this to be fixed, the resources CloudFront is using would need to be changed. Of the owners of the CDK doesn't have the capability to make that happen.

I think we need to be fair to everyone here, There is no right answer, only wrong answers, because the fact you can deploy CloudFront outside of us-east-1 but not the certificate for it is cursed.

[dickhardt]

@wparad - I agree with most of what you say -- but the true solution would be to enable CloudFront certs to be created in the same region as the CloudFront Distribution -- just like you can for a load balancer or other services that can take a cert.

Making me create my CloudFront Distribution in us-east-1 complicates things even more as it then requires all the rest of my stack that sits behind the distribution to either be in a different region or in us-east-1.

No resources should have to be in us-east-1. Requiring that breaks inter region isolation -- a best practice.

Unfortunately fixing that is not out of scope for the CDK team, so it needs to get escalated up the chain -- in the meantime we continue with the hack.

[blimmer]

I said this above, but I still feel that the real/best solution is to push forward with upstream CloudFormation support: aws-cloudformation/cloudformation-coverage-roadmap#523. That would remove all the hacky workarounds in CDK and it would just work ™️.

Please upvote that issue and, if you pay for an enterprise support plan with AWS, bring it up with your TAMs.

[pausethelogic]

@dickhardt The main difference between CloudFront and other services like ALBs that can also use ACM certificates is that CloudFront isn't regional - it exists in the Global region. All "global" AWS services (CloudFront, Route53, etc) are run out of us-east-1 on the backend.

ACM certificates can only be used in the region where they were created, like the majority of regional AWS services. That's why ACM certificates must exist in us-east-1 to be used by CloudFront, the same way an ALB running out of us-west-2 must use an ACM certificate that also exists in us-west-2.

That being said, I can see how having a single construct that can run in any region and automatically create a certificate in us-east-1 if using CloudFront could be helpful for some CloudFormation/CDK customers.

It seems like the main issue is that the current construct abstracts away the fact that CloudFront isn't regional, so users didn't have to think about it when creating distributions.

[dickhardt]

@blimmer agree solving it in CloudFront would then also solve it for CDK.
@pausethelogic I was a PE in AWS ... I understand :) ... but since I can create a CF distribution in any region and manage it there, I should also be able to do the same with the needed certificate, which can be created in any region already.