Merge pull request getlift#69 from getlift/single-table-construct

Single table construct

README.md

@@ -109,6 +109,16 @@ constructs:
 
 [Read more...](docs/webhook.md)
 
+### [Database - DynamoDB Single Table](docs/database-dynamodb-single-table.md)
+
+Deploy databases leveraging DynamoDB Single Table Design principles.
+
+```yaml
+constructs:
+ database:
+ type: database/dynamodb-single-table
+```
+
 More constructs are coming soon! Got suggestions? [Open and upvote drafts](https://github.com/getlift/lift/discussions/categories/components).
 
 ## Ejecting

docs/database-dynamodb-single-table.md

@@ -0,0 +1,115 @@
+# Database - DynamoDB Single Table
+
+The `database/dynamodb-single-table` construct deploys a single DynamoDB table with pre-configured indexes following [Single Table Design](https://www.alexdebrie.com/posts/dynamodb-single-table/) principles.
+
+## Quick start
+
+```yaml
+service: my-app
+provider:
+ name: aws
+
+constructs:
+ myTable:
+ type: database/dynamodb-single-table
+
+plugins:
+ - serverless-lift
+```
+
+On `serverless deploy`, a preconfigured DynamoDB table will be created.
+
+## How it works
+
+The `database/dynamodb-single-table` construct creates and configures the table for production following [Single Table Design](https://www.alexdebrie.com/posts/dynamodb-single-table/) principles:
+
+- a composite primary index with generic attributes names - `PK` for the partition key and `SK` for the sort key
+- a configurable amount of up to 20 glocal secondary indexes with generic names - `GSI-1` to `GSI-20` for the index names, `GSI-1-PK` to `GSI-20-PK` for the partition keys and `GSI-1-SK` to `GSI-20-SK` for the sort keys
+- all indexes attributes have string data type, ideal for composite attribue - i.e. `value1#value2`
+- a DynamoDB stream publishing new and old values at each write operation on the table
+- a TTL attribute enabling DynamoDB automatic garbage collection set to `TimeToLive`
+- a billing mode set to `PAY_PER_REQUEST`
+
+## Variables
+
+All database constructs expose the following variables:
+
+- `tableName`: the name of the deployed DynamoDB table
+- `tableStreamArn`: the ARN of the stream of the deployed DynamoDB table
+
+This can be used to inject the tableName to a Lambda functions using the SDK to read or write data from the table, for example:
+
+```yaml
+constructs:
+ myTable:
+ type: database/dynamodb-single-table
+
+functions:
+ myFunction:
+ handler: src/index.handler
+ environment:
+ TABLE_NAME: ${construct:myTable.tableName}
+```
+
+_How it works: the `${construct:myTable.tableName}` variable will automatically be replaced with a CloudFormation reference to the DynamoDB table._
+
+## Permissions
+
+By default, all the Lambda functions deployed in the same `serverless.yml` file **will be allowed to read/write into the table**, on all indexes (primary and secondary).
+
+In the example below, there are no IAM permissions to set up: `myFunction` will be allowed to read and write into the `myTable` table.
+
+```yaml
+constructs:
+ myTable:
+ type: database/dynamodb-single-table
+
+functions:
+ myFunction:
+ handler: src/index.handler
+ environment:
+ TABLE_NAME: ${construct:myTable.tableName}
+```
+
+## Configuration reference
+
+### Global secondary indexes
+
+Global secondary indexes have a direct impact on the cost of a DynamoDB table. There is no GSI configured by default on the database construct.
+
+You can specify the amount of GSI you'd like to enable on a DynamoDB table using the `gsiCount` property.
+
+```yaml
+constructs:
+ myTable:
+ # ...
+ gsiCount: 3
+```
+
+GSI created on the table follow generic names principles:
+- `GSI-1` to `GSI-20` for the index names
+- `GSI-1-PK` to `GSI-20-PK` for the partition keys
+- `GSI-1-SK` to `GSI-20-SK` for the sort keys
+
+The first time you deploy your construct using `serverless deploy`, you can specify any amount of GSI between `1` and `20`. On subsequent deploys, any modification made to an already deployed construct cannot add or remove more than 1 GSI at a time. If you need 2 additional GSI after initial deployment of the exemple above, you must first update the `gsiCount` to `4`, deploy, and then finally update it to the final desired quantity of `5`.
+
+### Local secondary indexes
+
+Each DynamoDB table can includes up to 5 local secondary indexes. You can deploy a table with those 5 indexes using the `localSecondaryIndexes` property.
+
+> :warning: LSIs introduce a [limitation on partition size of a table](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html#LSI.ItemCollections.SizeLimit). Due to this limitation, `localSecondaryIndexes` is set to false and this construct will not provision any LSI on the table by default.
+
+Setting `localSecondaryIndexes` to true will provision 5 LSIs with generic names - `LSI-1` to `LSI-5` for the index names and `LSI-1-SK` to `LSI-5-SK` for the sort keys. Those indexes have no impact on pricing as long as their sort keys are not populated with data.
+
+```yaml
+constructs:
+ myTable:
+ # ...
+ localSecondaryIndexes: true
+```
+
+> :warning: Modifying a table local secondary indexes configuration requires table re-creation. If you modify this setting after the table has been populated with data, you'll need to transfer all data from old table to the new one. You however won't loose any data as all tables are configured to be left as is when removed from a CloudFormation template.
+
+### More options
+
+Looking for more options in the construct configuration? [Open a GitHub issue](https://github.com/getlift/lift/issues/new).

package.json

@@ -11,6 +11,7 @@
  "@aws-cdk/aws-cloudfront": "^1.110.1",
  "@aws-cdk/aws-cloudfront-origins": "^1.110.1",
  "@aws-cdk/aws-cloudwatch": "^1.110.1",
+ "@aws-cdk/aws-dynamodb": "^1.110.1",
  "@aws-cdk/aws-ec2": "^1.110.1",
  "@aws-cdk/aws-events": "^1.110.1",
  "@aws-cdk/aws-iam": "^1.110.1",

src/classes/AwsProvider.ts

@@ -12,6 +12,7 @@ import { Queue } from "../constructs/Queue";
 import { Vpc } from "../constructs/Vpc";
 import { Webhook } from "../constructs/Webhook";
 import { StaticWebsite } from "../constructs/StaticWebsite";
+import { DatabaseDynamoDBSingleTable } from "../constructs/DatabaseDynamoDBSingleTable";
 
 export class AwsProvider {
  private static readonly constructClasses: Record<string, StaticConstructInterface> = {};
@@ -143,4 +144,4 @@ export class AwsProvider {
  * If they use TypeScript, `registerConstructs()` will validate that the construct class
  * implements both static fields (type, schema, create(), …) and non-static fields (outputs(), references(), …).
  */
-AwsProvider.registerConstructs(Storage, Queue, Webhook, StaticWebsite, Vpc);
+AwsProvider.registerConstructs(Storage, Queue, Webhook, StaticWebsite, Vpc, DatabaseDynamoDBSingleTable);

src/constructs/DatabaseDynamoDBSingleTable.ts

@@ -0,0 +1,107 @@
+import { Construct as CdkConstruct, CfnOutput, Fn, Stack } from "@aws-cdk/core";
+import { AttributeType, BillingMode, StreamViewType, Table } from "@aws-cdk/aws-dynamodb";
+import { FromSchema } from "json-schema-to-ts";
+import { AwsConstruct, AwsProvider } from "../classes";
+import { PolicyStatement } from "../CloudFormation";
+
+const DATABASE_DEFINITION = {
+ type: "object",
+ properties: {
+ type: { const: "database/dynamodb-single-table" },
+ localSecondaryIndexes: { type: "boolean" },
+ gsiCount: { type: "integer", minimum: 1, maximum: 20 },
+ },
+ additionalProperties: false,
+} as const;
+
+type Configuration = FromSchema<typeof DATABASE_DEFINITION>;
+const DATABASE_DEFAULTS: Required<Configuration> = {
+ type: "database/dynamodb-single-table",
+ localSecondaryIndexes: false,
+ gsiCount: 0,
+};
+
+export class DatabaseDynamoDBSingleTable extends AwsConstruct {
+ public static type = "database/dynamodb-single-table";
+ public static schema = DATABASE_DEFINITION;
+
+ private readonly table: Table;
+ private readonly tableNameOutput: CfnOutput;
+
+ constructor(scope: CdkConstruct, id: string, configuration: Configuration, private provider: AwsProvider) {
+ super(scope, id);
+
+ const resolvedConfiguration = Object.assign({}, DATABASE_DEFAULTS, configuration);
+
+ this.table = new Table(this, "Table", {
+ partitionKey: { name: "PK", type: AttributeType.STRING },
+ sortKey: { name: "SK", type: AttributeType.STRING },
+ billingMode: BillingMode.PAY_PER_REQUEST,
+ pointInTimeRecovery: true,
+ timeToLiveAttribute: "TimeToLive",
+ stream: StreamViewType.NEW_AND_OLD_IMAGES,
+ });
+
+ if (resolvedConfiguration.localSecondaryIndexes) {
+ for (let localSecondaryIndex = 1; localSecondaryIndex <= 5; localSecondaryIndex++) {
+ this.table.addLocalSecondaryIndex({
+ indexName: `LSI-${localSecondaryIndex}`,
+ sortKey: { name: `LSI-${localSecondaryIndex}-SK`, type: AttributeType.STRING },
+ });
+ }
+ }
+
+ if (resolvedConfiguration.gsiCount > 0) {
+ for (
+ let globalSecondaryIndex = 1;
+ globalSecondaryIndex <= resolvedConfiguration.gsiCount;
+ globalSecondaryIndex++
+ ) {
+ this.table.addGlobalSecondaryIndex({
+ indexName: `GSI-${globalSecondaryIndex}`,
+ partitionKey: { name: `GSI-${globalSecondaryIndex}-PK`, type: AttributeType.STRING },
+ sortKey: { name: `GSI-${globalSecondaryIndex}-SK`, type: AttributeType.STRING },
+ });
+ }
+ }
+
+ this.tableNameOutput = new CfnOutput(this, "TableName", {
+ value: this.table.tableName,
+ });
+ }
+
+ permissions(): PolicyStatement[] {
+ return [
+ new PolicyStatement(
+ [
+ "dynamodb:GetItem",
+ "dynamodb:BatchGetItem",
+ "dynamodb:Query",
+ "dynamodb:Scan",
+ "dynamodb:PutItem",
+ "dynamodb:DeleteItem",
+ "dynamodb:BatchWriteItem",
+ "dynamodb:UpdateItem",
+ ],
+ [this.table.tableArn, Stack.of(this).resolve(Fn.join("/", [this.table.tableArn, "index", "*"]))]
+ ),
+ ];
+ }
+
+ outputs(): Record<string, () => Promise<string | undefined>> {
+ return {
+ tableName: () => this.getTableName(),
+ };
+ }
+
+ variables(): Record<string, unknown> {
+ return {
+ tableName: this.table.tableName,
+ tableStreamArn: this.table.tableStreamArn,
+ };
+ }
+
+ async getTableName(): Promise<string | undefined> {
+ return this.provider.getStackOutput(this.tableNameOutput);
+ }
+}

test/fixtures/databasesDynamoDBSingleTable/serverless.yml

@@ -0,0 +1,15 @@
+service: storage
+configValidationMode: error
+
+provider:
+ name: aws
+
+constructs:
+ databaseWithoutSecondaryIndexes:
+ type: database/dynamodb-single-table
+ databaseWithLocalSecondaryIndexes:
+ type: database/dynamodb-single-table
+ localSecondaryIndexes: true
+ databaseWithGlobalSecondaryIndexes:
+ type: database/dynamodb-single-table
+ gsiCount: 2

test/unit/databasesDynamoDBSingleTable.test.ts

@@ -0,0 +1,113 @@
+import { pluginConfigExt, runServerless } from "../utils/runServerless";
+
+describe("databasesDynamoDBSingleTable", () => {
+ let cfTemplate: {
+ Resources: Record<string, { Properties: Record<string, unknown> }>;
+ Outputs: Record<string, unknown>;
+ };
+ let computeLogicalId: (...address: string[]) => string;
+ const tableUseCases = [
+ ["databaseWithoutSecondaryIndexes"],
+ ["databaseWithLocalSecondaryIndexes"],
+ ["databaseWithGlobalSecondaryIndexes"],
+ ];
+
+ beforeAll(async () => {
+ ({ cfTemplate, computeLogicalId } = await runServerless({
+ fixture: "databasesDynamoDBSingleTable",
+ configExt: pluginConfigExt,
+ command: "package",
+ }));
+ });
+ describe("common tests", () => {
+ test.each(tableUseCases)(
+ "%p - should ensure deletion policy and update replace policy are retain",
+ (tableUseCase) => {
+ expect(cfTemplate.Resources[computeLogicalId(tableUseCase, "Table")]).toMatchObject({
+ UpdateReplacePolicy: "Retain",
+ DeletionPolicy: "Retain",
+ });
+ }
+ );
+ test.each(tableUseCases)("%p - should provision generic names for primary index", (tableUseCase) => {
+ expect(
+ cfTemplate.Resources[computeLogicalId(tableUseCase, "Table")].Properties.AttributeDefinitions
+ ).toContainEqual({ AttributeName: "PK", AttributeType: "S" });
+ expect(
+ cfTemplate.Resources[computeLogicalId(tableUseCase, "Table")].Properties.AttributeDefinitions
+ ).toContainEqual({ AttributeName: "SK", AttributeType: "S" });
+ expect(cfTemplate.Resources[computeLogicalId(tableUseCase, "Table")].Properties.KeySchema).toEqual([
+ {
+ AttributeName: "PK",
+ KeyType: "HASH",
+ },
+ {
+ AttributeName: "SK",
+ KeyType: "RANGE",
+ },
+ ]);
+ });
+ });
+ it("should use generic names for LSI", () => {
+ for (let localSecondaryIndex = 1; localSecondaryIndex <= 5; localSecondaryIndex++) {
+ expect(
+ cfTemplate.Resources[computeLogicalId("databaseWithLocalSecondaryIndexes", "Table")].Properties
+ .AttributeDefinitions
+ ).toContainEqual({ AttributeName: `LSI-${localSecondaryIndex}-SK`, AttributeType: "S" });
+ }
+ expect(
+ cfTemplate.Resources[computeLogicalId("databaseWithLocalSecondaryIndexes", "Table")].Properties
+ .LocalSecondaryIndexes
+ ).toEqual(
+ Array.from({ length: 5 }, (_, i) => i + 1).map((localSecondaryIndex) => {
+ return {
+ IndexName: `LSI-${localSecondaryIndex}`,
+ KeySchema: [
+ {
+ AttributeName: "PK",
+ KeyType: "HASH",
+ },
+ {
+ AttributeName: `LSI-${localSecondaryIndex}-SK`,
+ KeyType: "RANGE",
+ },
+ ],
+ Projection: { ProjectionType: "ALL" },
+ };
+ })
+ );
+ });
+ it("should use generic names for GSI", () => {
+ for (let globalSecondaryIndex = 1; globalSecondaryIndex <= 2; globalSecondaryIndex++) {
+ expect(
+ cfTemplate.Resources[computeLogicalId("databaseWithGlobalSecondaryIndexes", "Table")].Properties
+ .AttributeDefinitions
+ ).toContainEqual({ AttributeName: `GSI-${globalSecondaryIndex}-PK`, AttributeType: "S" });
+ expect(
+ cfTemplate.Resources[computeLogicalId("databaseWithGlobalSecondaryIndexes", "Table")].Properties
+ .AttributeDefinitions
+ ).toContainEqual({ AttributeName: `GSI-${globalSecondaryIndex}-SK`, AttributeType: "S" });
+ }
+ expect(
+ cfTemplate.Resources[computeLogicalId("databaseWithGlobalSecondaryIndexes", "Table")].Properties
+ .GlobalSecondaryIndexes
+ ).toEqual(
+ Array.from({ length: 2 }, (_, i) => i + 1).map((globalSecondaryIndex) => {
+ return {
+ IndexName: `GSI-${globalSecondaryIndex}`,
+ KeySchema: [
+ {
+ AttributeName: `GSI-${globalSecondaryIndex}-PK`,
+ KeyType: "HASH",
+ },
+ {
+ AttributeName: `GSI-${globalSecondaryIndex}-SK`,
+ KeyType: "RANGE",
+ },
+ ],
+ Projection: { ProjectionType: "ALL" },
+ };
+ })
+ );
+ });
+});