Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Multi-Region Access Point(MRAP) is not available unless with additional dependency #2822

Open
AllanZhengYP opened this issue Sep 24, 2021 · 22 comments
Assignees
Labels
feature-request New feature or enhancement. May require GitHub community feedback. p2 This is a standard priority issue queued This issues is on the AWS team's backlog

Comments

@AllanZhengYP
Copy link
Contributor

AllanZhengYP commented Sep 24, 2021

Description

Package @aws-sdk/client-s3 launched the support for S3 Multi-Region Access Point (MRAP) feature in v3.31.0. This feature relies on signer class from aws-crt package. This dependency requires native addons that takes up to 10.5 MB on disk. Because the dependency is only required by MRAP feature, we are omitting the dependency in @aws-sdk/client-s3.

To use this feature, you need to install this dependency explicitly in your application:

$ npm install @aws-sdk/signature-v4-crt
$ yarn add @aws-sdk/signature-v4-crt

Because aws-crt contains platform-specific binary code, the following use cases are not supported in MRAP:

  • browsers
  • React Native
  • bundle as Node.js application

The suggested use case for supporting MRAP in Node.js app is to deploy your application with the complete node_modules folder.

Describe alternatives you've considered

We are working on solutions to unblock the MRAP for all the supported platforms.

Related issues:

@AllanZhengYP AllanZhengYP added the feature-request New feature or enhancement. May require GitHub community feedback. label Sep 24, 2021
@AllanZhengYP AllanZhengYP pinned this issue Sep 25, 2021
@seriousme
Copy link

seriousme commented Oct 16, 2021

FYI: One might consider porting the signer class of the common runtime to webassembly. Web assembly is supported in all major browsers and in NodeJs 8+ See also: https://developer.mozilla.org/en-US/docs/WebAssembly/C_to_wasm

Kind regards,
Hans

@AllanZhengYP
Copy link
Contributor Author

Thank you for the advice! @seriousme

We are currently weigh in the options including WASM, ASM.js, and pure JS with WebCrypto. Currently one of the obstacle is that WASM seems not supported in all React Native platforms, so we might need to fallback to use ASM.js in React Native.

On the other hand, we need to make sure the signer is cryptographically secure. If the implementation involves 3rd party crypto primitives, we need to audit them. For example, the signing algorithm in SigV4a requires constant time implementation.

@seriousme
Copy link

Good to see you have options :-)

You can mix and match, e.g. use the browsers webcrypto from WASM ;-)
e.g. https://ihsavru.medium.com/calling-javascript-code-from-c-c-using-webassembly-a9445c11bc6d

Kind regards,
Hans

@christophgysin
Copy link
Contributor

Adding aws-crt blocks me from updating the deno port of aws-sdk-js-v3 past v3.33.0.

👍 for WASM, which would also allow using this in deno.

@rossng
Copy link

rossng commented Jan 26, 2022

Just want to weigh in with the impact this issue currently has on our team. Hopefully it helps to motivate a fix. Our developers use a mix of macOS and Windows machines.

We depend on the @aws-sdk/client-s3 package. This causes the package manager to complain about the missing @aws-sdk/signature-v4-crt peer dependency.

The Windows users go ahead and add the dep to package.json, and everything appears to be resolved. Then a macOS user pulls the latest changes and suddenly package installation is completely broken.

This isn't a great experience. Given that @aws-sdk/signature-v4-crt has ~63k downloads per week, I can only imagine how many developers have run headlong into this so far.

@jekh
Copy link

jekh commented Mar 17, 2022

I just ran into this issue, seeing the same peer dependency warnings as @rossng. Using the peerDependenciesMeta field of package.json might be a good solution: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#peerdependenciesmeta

Adding this to package.json in middleware-sdk-s3 should fix this:

"peerDependenciesMeta": {
   "@aws-sdk/signature-v4-crt": {
     "optional": true
   }
 }

@AllanZhengYP
Copy link
Contributor Author

@jekh Thanks a lot for pointing this out! It's a miss on our side. I will address them a separate PR.

@CraigStuntzWillowTree
Copy link

I just tried this with 3.58.0 and I'm still seeing the error on macOS:

WARNING in ../node_modules/@aws-sdk/util-user-agent-node/dist-es/is-crt-available.js 3:96-114
Module not found: Error: Can't resolve 'aws-crt' in '/Users/craig/Documents/PMI/node_modules/@aws-sdk/util-user-agent-node/dist-es'
 @ ../node_modules/@aws-sdk/util-user-agent-node/dist-es/index.js 5:0-52 16:23-37
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/runtimeConfig.js 12:0-65 22:898-914
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/SSMClient.js 10:0-73 15:24-42
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/index.js 2:0-28 2:0-28
...

...where I have the library referenced like so:

    "@aws-sdk/client-ssm": "^3.58.0"

and it's warning on this line:

import { SSM } from "@aws-sdk/client-ssm";

@Hasan-Jawaheri
Copy link

This issue is affecting our team and our entire organization will run into the same issue soon as we are moving towards a DNS regional failover which requires SigV4A signing in the browser to make API calls to unkown AWS region (using @aws-sdk/signature-v4-crt).

Are there alternatives for the time being? Can we do SigV4A signing with existing parts of the SDK that work on browser?

@samthompsonkennedy
Copy link

This is insane.
No update from Amazon? This has put an entire build on hold.

Minimal documentation for MRAP as it is.

@ajredniwja ajredniwja added the wontfix We have determined that we will not resolve the issue. label Sep 14, 2022
@RanVaknin RanVaknin unpinned this issue Jan 3, 2023
@trivikr trivikr pinned this issue Jan 3, 2023
@ashley-jisc
Copy link

Still nothing for this?

@JClackett
Copy link

any updates on this?

@anta-semenov
Copy link

anta-semenov commented Apr 19, 2023

any updates on this?

We're trying to use apigateway in multiple regions with custom domains with iam authorization.
The request can be routed to different region based on latency, so regular sigV4 doesn't work.

Is there any other workaround to make apigateway works in multiple regions with the same domain name?

@anta-semenov
Copy link

@AllanZhengYP I'm curious does c++ code has any platform specific dependencies?
It seems like the latest version of react native has first class support c++ code with turbomodules. Does the team consider the option to create a separate rn library for that

@summaarum
Copy link

Would also like like updates on this. Getting error during build Module not found: Can't resolve '@aws-sdk/signature-v4-crt' on latest version 3.354.0

@MRdNk
Copy link

MRdNk commented Aug 17, 2023

I'm getting the following error in CloudWatch, when trying to do a putObjectCommand to an MRAP...

ERROR	Invoke Error 	
{
    "errorType": "TypeError",
    "errorMessage": "Cannot read properties of undefined (reading 'enable_logging')",
    "$metadata": {
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "TypeError: Cannot read properties of undefined (reading 'enable_logging')",
        "    at new CrtSignerV4 (/var/task/[bundleName].js:41108:22)",
        "    at SignatureV4MultiRegion.getSigv4aSigner (/var/task/[bundleName].js:41056:33)",
        "    at SignatureV4MultiRegion.sign (/var/task/announcementHandlerBundle.js:41029:25)",
        "    at /var/task/[bundleName].js:9155:31",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/task/[bundleName].js:6896:46",
        "    at async /var/task/[bundleName].js:54832:20",
        "    at async /var/task/[bundleName].js:5636:26"
    ]
}

Using the latest S3 and cert lib., in my package.json ...

"@aws-sdk/client-s3": "^3.391.0",
"@aws-sdk/signature-v4-crt": "^3.391.0",

S3Client setup as follows (in side a Ts Lambda function, which works if I just use a standard s3 bucket):

const s3Client = new S3Client({ useGlobalEndpoint: true });

I've tried with and without useGlobalEndpoint; as well as with and without setting the region: 'eu-west-2'; the region the lambda is running in - all the same issue.

NB. Bucket name is set as the MRAP:
arn:aws:s3::123456789:accesspoint/[alias].mrap

Any help appreciated.

@MRdNk
Copy link

MRdNk commented Aug 18, 2023

Just read the doesn’t work with bundled nodejs applications; with advice to copy in the node_modules.
That’s not going to be easy

@MicrosoftJones
Copy link

Patch

@kuhe
Copy link
Contributor

kuhe commented Sep 18, 2023

The AWS SDK for JavaScript team is planning to change the usage of the optional CRT dependency in a future update.

The change is described in this announcement: #5229.

The change is aimed at reducing errors encountered by bundlers and other static analysis tools. The @aws-sdk/signature-v4-crt package remains optional and is only required when using S3 MRAP or EventBridge.

@Legacy131313
Copy link

Description

Package @aws-sdk/client-s3 launched the support for S3 Multi-Region Access Point (MRAP) feature in v3.31.0. This feature relies on signer class from aws-crt package. This dependency requires native addons that takes up to 10.5 MB on disk. Because the dependency is only required by MRAP feature, we are omitting the dependency in @aws-sdk/client-s3.

To use this feature, you need to install this dependency explicitly in your application:

$ npm install @aws-sdk/signature-v4-crt
$ yarn add @aws-sdk/signature-v4-crt

Because aws-crt contains platform-specific binary code, the following use cases are not supported in MRAP:

  • browsers
  • React Native
  • bundle as Node.js application

The suggested use case for supporting MRAP in Node.js app is to deploy your application with the complete node_modules folder.

Describe alternatives you've considered

We are working on solutions to unblock the MRAP for all the supported platforms.

Related issues:

@kuhe kuhe unpinned this issue Oct 25, 2023
@oskbor
Copy link

oskbor commented Feb 9, 2024

So I have spent the last weeks setting up buckets, replication rules, an Multi Region Access Point and I have gotten STS to sign an access token that I have now passed to my client. And here I am, just now finding out that oh btw, multi region access points cannot be queried from the browser. Que? I'm fuming right now.

Isn't that like a very common use case to query s3 from the browser?
At least put this information in some MRAP docs 😢 Very sad to have spent so much time on this.

Please tell me that there is a way to do GetObject from a browser via an mrap.

@zhuming
Copy link

zhuming commented Feb 23, 2024

anywhere I can find a sample code or documentation how to use @aws-sdk/signature-v4-crt to sign an http request sent to aws service. I'm using aws opensearch client, but it has a gap in covering all apis supported by opensearch (elasticSearch), compared to elasticSearchClient. So I can either switch to elasticSearch client, or keep using openSearchClient, but with a polyfill of sending a few signed http request directly to opensearch service. Either way, I need to work on this signing request. I used to use v2 to sign elasticSearch client request, but I believe it's deprecated now. Any suggestions?

@aBurmeseDev aBurmeseDev added the queued This issues is on the AWS team's backlog label Mar 6, 2024
@aBurmeseDev aBurmeseDev removed the wontfix We have determined that we will not resolve the issue. label Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request New feature or enhancement. May require GitHub community feedback. p2 This is a standard priority issue queued This issues is on the AWS team's backlog
Projects
None yet
Development

Successfully merging a pull request may close this issue.