-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SagemakerModel.transform() doesn't use model's sagemakerClient #28
Comments
Hi @harthur, Thanks for using Amazon SageMaker! There are two SageMaker clients: the Instead of injecting this client, you have to change the value of this var in the
Why? Because Spark has to serialize tasks to send them to workers in the Please feel free to reopen this if it doesn't answer your question. Thanks! |
That does answer my question, thanks! However, how do you set Let me know if this isn't the right place to ask more questions. |
Hey @harthur, I'm not sure exactly when the client is run. It's possible we should make that a Otherwise: you're right about the client builder -- to get around that error, you can set AWS_DEFAULT_REGION to your region (like us-west-2 or us-east-1 or us-east-2 or eu-west-1, any of the SageMaker regions) or set it in your AWS config file with Thanks! |
The stacktrace is
Where that line is: RequestBatchIterator.sagemakerRuntime = sagemakerRuntimeClient |
Hi @harthur , Thanks for the stacktrace! Just FYI: I haven't gotten a chance to reproduce this yet, but this definitely seems like a bug. I suppose that workers are still trying to create the standard client. If you're able to, could you post your code? Otherwise, to unblock yourself in the short term, it seems like you'll have to get the region from the environment in your workers (by setting Thanks again! |
Yeah, I got around it by adding some Java system properties, but ideally you would be able to build your own client and keep that info isolated. That's the pattern we use for all of our other AWS connections, so it's a bit awkward to break that just for Sagemaker. |
Sorry, to address your first question, I think it's happening in the driver rather than on any workers (or, that's where this particular line of code is). It happens just by instantiating |
Hey @harthur , Ah, interesting, thanks for the update! Glad to hear you got it working, but you're right, we should let users build their own client. I've put a fix for this on our backlog, thanks for reporting this. We'll keep this issue open and update this when the fix is in, but I can't give an ETA on when we'll be able to do this. Thanks again! |
Hi @harthur , Thanks |
@sreemani You have to set these system properties: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default |
Any update on this issue? This bug has been open since 2018, and it is causing our team some problems. |
If you create a new
SageMakerModel
instance (say, withfromModelS3Path()
), you can pass in your ownsagemakerClient
. However, when you go to use the model after it's been created,transform()
does not use that client to send prediction requests. It appears to hardcode aAmazonSageMakerRuntimeClientBuilder.defaultClient
instead inRequestBatchIterator
.Pardon my ignorance, but is there a reason that it can't just pass the
sagemakerClient
through?The text was updated successfully, but these errors were encountered: