feat: Generate DDB client code for Java V2 AWS SDK #92

lucasmcdonald3 · 2023-01-25T21:59:10Z

Note this is based on another change that is currently in PR: #82

#82 makes changes that are used in this PR. #82 should be merged into main-1.x before this change is merged.

Description of changes:

Generate DDB code for AWS SDK for Java V2

Testing:

I ran bin/generate_code.sh from my private-ESDK-dafny repo, then ran make transpile_java && make test_java.

The tests run GetItem, QueryItem, and PutItem. See test file.

I opened a draft PR with the generated changes in my fork of private-ESDK-dafny: link

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

texastony · 2023-02-13T22:35:51Z

...rph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToDafnyAwsV2.java

+ // This is the only time when Polymorph needs to convert a list of a Dafny type to a list
+ // of a type that Polymorph does not know about. So this is a special case and warrants
+ // its own generation logic.
+ if (targetShape.getId().getName().equals("BinarySetAttributeValue")) {


Issue: I am a little hesitant on using only the name of a shape to trigger an edge case.
BinarySetAttributeValue is probably fine, as it's so unique,
no other AWS Service model we use will include it...
Actually, another Database Service (RDS, Aurora, etc. ) might...

texastony · 2023-02-13T22:38:42Z

...rph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToDafnyAwsV2.java

+ // SizeEstimateRangeGB is a list of the case above, where Smithy models values as integers,
+ // but the SDK expects doubles.
+ // This is the only instance of double -> int conversion in a list right now.
+ if (memberShape.getMemberName().equals("SizeEstimateRangeGB")) {


Issue: Many AWS Services could have a shape named SizeEstimateRangeGB.
We only know this logic works for DDB...
We should use both the namespace and the name to protect this edge case.

texastony · 2023-02-13T22:39:31Z

...rph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToDafnyAwsV2.java

+ return shapeName.equals("ConsumedCapacityUnits")
+ || shapeName.equals("Double");


Issue: Many AWS Services could have a shape named ConsumedCapacityUnits.
We only know this logic works for DDB...
We should use both the namespace and the name to protect this edge case.

texastony · 2023-02-13T22:41:52Z

...rph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToDafnyAwsV2.java

+ .addAnnotation(
+ AnnotationSpec.builder(SuppressWarnings.class)
+ .addMember("value", "$S", "unchecked")
+ .build()


Praise: Nice job on figuring this out!
Nuances with Dafny like this trip my polymorph work up all the time.

texastony · 2023-02-13T22:44:04Z

smithy-polymorph/src/main/java/software/amazon/polymorph/smithyjava/generator/Generator.java

+ public static final ClassName BLOB_TO_NATIVE_SDK_BYTES =
+ ClassName.get("software.amazon.awssdk.core", "SdkBytes");


Issue/Suggestion: It may be best if this constant lived in awssdk/v2/ShimV2.java,
as compared to here.

texastony · 2023-02-13T22:46:59Z

...ph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToNativeAwsV2.java

+ // "TargetValue" refers to a table's target R/W capacity target values.
+ // SDK handles these as doubles, but the Smithy model stores them as integers.
+ if (member.getMemberName().equals("TargetValue")) {


Issue: Many AWS Services could have a shape named TargetValue.
We only know this logic works for DDB...
We should use both the namespace and the name to protect this edge case.

...ph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToNativeAwsV2.java

texastony · 2023-02-13T22:51:56Z

smithy-polymorph/src/main/java/software/amazon/polymorph/smithyjava/nameresolver/Native.java

+ public ClassName classForDouble() {
+ return ClassName.get(Double.class);
+ }


Question: I wonder why we need this method?

texastony · 2023-02-13T22:56:44Z

...ymorph/src/main/java/software/amazon/polymorph/smithyjava/nameresolver/v2/AwsSdkDafnyV2.java

 if ("message".equals(uncapitalize(memberShape.getMemberName()))) {
- return CodeBlock.of("$L.get$L()", variableName, capitalize(memberShape.getMemberName()));
+ // BatchStatementError and CancellationReason types' messages are retrieved via "message".
+ if (memberShape.getContainer().getName().contains("BatchStatementError")
+ || memberShape.getContainer().getName().contains("CancellationReason")) {
+ return CodeBlock.of("$L.$L()", variableName,
+ uncapitalize(memberShape.getMemberName()));


Issue: This follows from my comment on the KMS PR.
#82 (comment)
We should go ahead and be specific here.
Fully qualify BatchStatementError and CancellationReason
(both are names that could be in any AWS Service Model, but not get treated the same as DDB's).
I'd also advocate for handling Error Structures seperately...
i.e.: if the member name is message and it's structure has the error trait, use getMessage).

texastony · 2023-02-13T22:59:14Z

...ymorph/src/main/java/software/amazon/polymorph/smithyjava/nameresolver/v2/AwsSdkDafnyV2.java

+ public boolean shapeIdRequiresStaticTypeDescriptor(final ShapeId shapeId) {
+ String className = classForNotErrorNotUnitShape(shapeId).toString();
+
+ return (className.equals("Dafny.Com.Amazonaws.Dynamodb.Types.AttributeMap")
+ || className.equals("Dafny.Com.Amazonaws.Dynamodb.Types.Key"));


Praise: This is great and safe as the edge case is handled by a fully qualified name.

texastony · 2023-02-13T23:02:29Z

...morph/src/main/java/software/amazon/polymorph/smithyjava/nameresolver/v2/AwsSdkNativeV2.java

+ if (smithyName.simpleName().endsWith("Input")) {
+ return ClassName.get(smithyName.packageName(),
+ smithyName.simpleName()
+ .substring(
+ 0,
+ smithyName.simpleName().lastIndexOf("Input"))
+ + "Request"
+ );
+ }
+ if (smithyName.simpleName().endsWith("Output")) {
+ return ClassName.get(smithyName.packageName(),
+ smithyName.simpleName()
+ .substring(
+ 0,
+ smithyName.simpleName().lastIndexOf("Output"))
+ + "Response"
+ );


Question: Why did we not need this for KMS?
Is this not true for KMS?
Or do all KMS shapes already use Request and Response... I bet that is it...

texastony · 2023-02-14T23:18:10Z

...rph/src/main/java/software/amazon/polymorph/smithyjava/generator/awssdk/v2/ToDafnyAwsV2.java

+ // OperationErrors are streamed to gather the types that are only inputs or outputs to errors.
+ // Right now this is only CancellationReasonList.
+ LinkedHashSet<ShapeId> operationErrors = subject.serviceShape
+ .getOperations().stream()
+ .map(shapeId -> subject.model.expectShape(shapeId, OperationShape.class))
+ .map(OperationShape::getErrors)
+ .flatMap(Collection::stream)
+ .filter(structureShape -> !structureShape.toString().contains("InvalidEndpointException"))
+ .collect(Collectors.toCollection(LinkedHashSet::new));


I see the value in this.
But, operationErrors is left in allReleventShapeIds.
So... those Error structures will get included in convertAllRelevent,
and will be passed to generateConvertStructure.
Then, as part of convertServiceErrors, they will get
another conversion method, but a modeledError conversion method.
I will refactor this so that we only generate modeledError.

texastony · 2023-02-15T01:55:39Z

...ymorph/src/main/java/software/amazon/polymorph/smithyjava/nameresolver/v2/AwsSdkDafnyV2.java

+ if (memberShape.getContainer().getName().contains("BatchStatementError")
+ || memberShape.getContainer().getName().contains("CancellationReason")) {


Issue: BatchStatementError (link to shape in model) and
CancellationReason (link to shape in model) are not marked with the @error trait.

Thus, they are just normal structures.
That is why their access is via message, as compared to getMessage.
In Java, getMessage is method of java.lang.Exception,
the global base Exception.

All AWS SDK exceptions extend from java.lang.Exception,
as do the Exceptions we generate.

This is why message is accessed by getMessage for structures marked with the Error trait.

Where "srv's Op strucs" is a Service's Operation Structures, or all of a Service's Operations' Input, Output, Errors and all shapes related to the above. We need dafny-lang/dafny#3034 to resolve singular datatype creation (now known as a "Record Type"). DONE: - [x] Test Java KMS - [x] Test Java DDB - [x] Merge #83 into this TODO: - [ ] Test Java DDB again - [ ] Address Issues raised in PR #82, #92, #83

Use the codegen for the AWS SDK V2 to correctly name the Enums. I have also addressed some issues raised in the PRs. DONE: - [x] Test Java KMS - [x] Test Java DDB - [x] Merge #83 into this TODO: - [ ] Implement ToNative Double solution - [ ] Test Java DDB again - [ ] Address Issues raised in PR #82, #92, #83 - [ ] Consolidate duplicated code - [ ] Test Java KMS & DDB Again - [ ] PR & Merge

TODO: - [ ] Implement ToNative Double solution - [ ] Test Java DDB again - [ ] Address Issues raised in PR #82, #92, #83 - [ ] Consolidate duplicated code - [ ] Test Java KMS & DDB Again - [ ] PR & Merge

Where "srv's Op strucs" is a Service's Operation Structures, or all of a Service's Operations' Input, Output, Errors and all shapes related to the above. We need dafny-lang/dafny#3034 to resolve singular datatype creation (now known as a "Record Type"). DONE: - [x] Test Java KMS - [x] Test Java DDB - [x] Merge #83 into this TODO: - [ ] Test Java DDB again - [ ] Address Issues raised in PR #82, #92, #83

Use the codegen for the AWS SDK V2 to correctly name the Enums. I have also addressed some issues raised in the PRs. DONE: - [x] Test Java KMS - [x] Test Java DDB - [x] Merge #83 into this TODO: - [ ] Implement ToNative Double solution - [ ] Test Java DDB again - [ ] Address Issues raised in PR #82, #92, #83 - [ ] Consolidate duplicated code - [ ] Test Java KMS & DDB Again - [ ] PR & Merge

TODO: - [ ] Implement ToNative Double solution - [ ] Test Java DDB again - [ ] Address Issues raised in PR #82, #92, #83 - [ ] Consolidate duplicated code - [ ] Test Java KMS & DDB Again - [ ] PR & Merge

seebees · 2023-03-31T17:22:24Z

This landed elsewhere right?

texastony · 2023-03-31T20:17:47Z

This landed elsewhere right?

Yes. I merged this else where. I left this open as I wanted to make sure I came back are resolved these issues...
But that is now all tracked in https://sim.amazon.com/issues/CrypTool-5101

Create a shared makefile, and then use that shared makefile

Lucas McDonald added 19 commits December 23, 2022 10:17

Remove unions

9219a7d

[DO NOT MERGE] WIP for DDB with SDK V2

74b1adf

Merge branch 'main-1.x' into java-sdk-v2-ddb

fa80c44

feat: Generate correctly for ConsumedCapacityUnits

3f9b661

feat: Generate sizeEstimateRangeGB correctly

e56c6b7

fix: Generate ConsumedCapacityUnits code with better logical trigger

9322cf5

Builds without extra changes

652049f

Cleanup

9948c78

Cleanup

2f6e547

Cleanup

da0af47

Cleanup

73c55d8

Cleanup

5a1c319

Fix models missing predicates building

31ee5db

Fix extra underscore, cleanup

4209242

Merge branch 'main-1.x' into java-sdk-v2-ddb

3706a01

Merge branch 'java-sdk-v2-kms' into java-sdk-v2-ddb

e995ae7

Cleanup

edafd32

Refactor

072ba4e

Refactor

f1028cf

lucasmcdonald3 requested a review from a team as a code owner January 25, 2023 21:59

texastony reviewed Feb 13, 2023

View reviewed changes

texastony mentioned this pull request Feb 13, 2023

Issue: Does Gazelle need Double Support #140

Closed

texastony reviewed Feb 14, 2023

View reviewed changes

texastony reviewed Feb 15, 2023

View reviewed changes

texastony closed this Mar 31, 2023

lucasmcdonald3 deleted the java-sdk-v2-ddb branch August 15, 2023 22:20

robin-aws pushed a commit that referenced this pull request Apr 19, 2024

feat: Consolidate make targets (#92)

6d328d7

Create a shared makefile, and then use that shared makefile

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Generate DDB client code for Java V2 AWS SDK #92

feat: Generate DDB client code for Java V2 AWS SDK #92

lucasmcdonald3 commented Jan 25, 2023 •

edited

Loading

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 13, 2023

texastony Feb 14, 2023

texastony Feb 15, 2023

seebees commented Mar 31, 2023

texastony commented Mar 31, 2023

		return shapeName.equals("ConsumedCapacityUnits")
		\|\| shapeName.equals("Double");

		public static final ClassName BLOB_TO_NATIVE_SDK_BYTES =
		ClassName.get("software.amazon.awssdk.core", "SdkBytes");

		if (memberShape.getContainer().getName().contains("BatchStatementError")
		\|\| memberShape.getContainer().getName().contains("CancellationReason")) {

feat: Generate DDB client code for Java V2 AWS SDK #92

feat: Generate DDB client code for Java V2 AWS SDK #92

Conversation

lucasmcdonald3 commented Jan 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seebees commented Mar 31, 2023

texastony commented Mar 31, 2023

lucasmcdonald3 commented Jan 25, 2023 •

edited

Loading