Skip to content

weka/terraform-aws-weka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

terraform-aws-weka

AWS terraform weka deployment module.
Applying this terraform module will create the following resources:

  • DynamoDB table (stores the the weka cluster state default KMS key)

  • Lambda:

    • deploy - responsible for providing new machines installation script

    • clusterize - responsible for providing clusterize script

    • clusterize-finalization - responsible for updating the cluster state about clusterization completion

    • report - responsible for updating the state about clusterization and new machines installation progress

    • status - responsible for providing the cluster progress status

    • for State Machine:

      • fetch - fetches cluster/autoscaling group information and passes to the next stage
      • scale-down - relied on fetch information to work on the Weka cluster, i.e., deactivate drives/hosts. Will fail if the required target is not supported (like scaling down to 2 backend instances)
      • terminate - terminates deactivated hosts
      • transient - lambda responsible for reporting transient errors, e.g., could not deactivate specific hosts, but some have been deactivated, and the whole flow proceeded
  • Launch Template: used for new auto-scaling group instances; will run the deploy script on launch.

  • Ec2 instances

  • Placement Group

  • Auto Scaling Group

  • ALB (optional for UI and Backends)

  • State Machine: invokes the fetch, scale-down, terminate, transient

    • Uses the previous lambda output as input for the following lambda.
    • CloudWatch: invokes the state machine every minute
  • SecretManager (stores the weka user name, password and get.weka.io token)

  • IAM Roles (and policies):

Weka deployment prerequisites:

  • vpc (with secret manager endpoint)

  • subnet (optional: additional_alb_subnet for ALB)

  • security group (with self reference rule)

  • iam roles

    Ec2 iam policy (replace *prefix* and *cluster_name* with relevant values)
    {
        "Statement": [
        {
            "Action": [
                "ec2:DescribeNetworkInterfaces",
                "ec2:AttachNetworkInterface",
                "ec2:CreateNetworkInterface",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:DeleteNetworkInterface"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "lambda:InvokeFunction"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:lambda:*:*:function:prefix-cluster_name*"
            ]
        },
        {
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::prefix-cluster_name-obs/*"
            ]
        },
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "logs:PutRetentionPolicy"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:logs:*:*:log-group:/wekaio/prefix-cluster_name*"
            ]
        }
        ],
        "Version": "2012-10-17"
    }
    Lambda iam policy (replace *prefix* and *cluster_name* with relevant values)
    {
        "Statement": [
        {
            "Action": [
              "s3:CreateBucket"
            ],
            "Effect": "Allow",
            "Resource": [
              "arn:aws:s3:::prefix-cluster_name-obs"
            ]
        },
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow",
            "Resource": [
              "arn:aws:logs:*:*:log-group:/aws/lambda/prefix-cluster_name*:*"
            ]
        },
        {
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface",
                "ec2:ModifyInstanceAttribute",
                "ec2:TerminateInstances",
                "ec2:DescribeInstances",
                "ec2:CreateTags"
            ],
            "Effect": "Allow",
            "Resource": [
              "*"
            ]
        },
        {
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:UpdateItem"
            ],
            "Effect": "Allow",
            "Resource": [
              "arn:aws:dynamodb:*:*:table/prefix-cluster_name-weka-deployment"
            ]
        },
        {
            "Action": [
              "secretsmanager:GetSecretValue",
              "secretsmanager:PutSecretValue"
            ],
            "Effect": "Allow",
            "Resource": [
              "arn:aws:secretsmanager:*:*:secret:weka/prefix-cluster_name/*"
            ]
        },
        {
            "Action": [
                "autoscaling:DetachInstances",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:SetInstanceProtection"
            ],
            "Effect": "Allow",
            "Resource": [
              "*"
            ]
        },
        {
          "Action": [
            "lambda:InvokeFunction"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:lambda:*:*:function:prefix-cluster_name-*-lambda"
          ]
        }
        ],
        "Version": "2012-10-17"
        }
    State Machine iam policy (replace *prefix* and *cluster_name* with relevant values)
    {
      "Statement": [
        {
          "Action": [
            "lambda:InvokeFunction"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:lambda:*:*:function:prefix-cluster_name-*-lambda"
          ]
        },
        {
          "Action": [
            "logs:CreateLogDelivery",
            "logs:GetLogDelivery",
            "logs:UpdateLogDelivery",
            "logs:DeleteLogDelivery",
            "logs:ListLogDeliveries",
            "logs:PutLogEvents",
            "logs:PutResourcePolicy",
            "logs:DescribeResourcePolicies",
            "logs:DescribeLogGroups"
          ],
          "Effect": "Allow",
          "Resource": [
            "*"
          ]
        }
      ],
      "Version": "2012-10-17"
    }
    Cloud Watch Events iam policy (replace *prefix* and *cluster_name* with relevant values)
    {
      "Statement": [
        {
          "Action": [
            "states:StartExecution"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:states:*:*:stateMachine:prefix-cluster_name-scale-down-state-machine"
          ]
        }
      ],
      "Version": "2012-10-17"
    }

Usage example:

This example will automatically create a vpc, subnets, security group and iam roles.

provider "aws" {
}

module "deploy_weka" {
  source                            = "weka/weka/aws"
  version                           = "1.0.1"
  prefix                            = "weka-tf"
  cluster_name                      = "test"
  allow_ssh_cidrs                   = ["0.0.0.0/0"]
  get_weka_io_token                 = "..."
}

output "deploy_weka_output" {
  value = module.deploy_weka
}

Using existing vpc:

vpc_id                            = "..."

Using existing subnet:

subnet_ids                        = ["..."]

Using existing security groups:

sg_ids                            = ["..."]

Using existing iam roles:

instance_iam_profile_arn          = "..."
lambda_iam_role_arn               = "..."
sfn_iam_role_arn                  = "..."
event_iam_role_arn                = "..."

Helper modules

We provide iam, network and security_group modules to help you create the prerequisites for the weka deployment.
Check our example that uses these modules.

  • When sg_ids isn't provided we automatically create a security group using our module.
  • When subnet_ids isn't provided we automatically create a subnet using our module.
  • When instance_iam_profile_arn isn't provided we automatically create an iam instance profile using our module.
  • var availability_zones need to provide only when we create network module, Currently limited to single subnet. for example eu-west-1c

NAT network deployment:

we provide module for creating private network with NAT To create private vpc with NAT, you must provide the following variables:

create_nat_gateway      = true
nat_public_subnet_cidr = PUBLIC_CIDR_RANGE

Private network deployment:

we provide module for creating private network with NO internet access To create private vpc, you must provide the following variables:

subnet_autocreate_as_private = true

To avoid public ip assignment:

assign_public_ip   = false

Ssh keys

The username for ssh into vms is ec2-user. If ami_id is provided by the user, the default ssh username will be accordingly.

We allow passing existing key pair name:

key_pair_name = "..."

We allow passing an existing public key string to create new key pair:

ssh_public_key = "..."

If key pair name and public key aren't passed we will create it for you and store the private key locally under /tmp Names will be:

/tmp/${prefix}-${cluster_name}-public-key.pub
/tmp/${prefix}-${cluster_name}-private-key.pem

To disable using key pair need to set:

enable_key_pair = false

To pass any custom data to init script, for example to install SSM need to set:

custom_data = "sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm\n sudo systemctl start amazon-ssm-agent"

Create ALB

We support ALB creation for backend UI, and joining weka clients will use this ALB to join a cluster, allowing for better distribution of load amongst backends. mandatory variables you must provide are:

create_alb                       = true
alb_additional_subnet_cidr_block = ADDITIONAL_SUBNET_CIDR_BLOCK

To use existing additional subnet, you must supply the following variables:

additional_alb_subnet_id = SUBNET_ID
alb_sg_ids               = ALB_SG_IDS

To add ALB dns name to zone record, you must supply the following variables:

alb_alias_name      = ALB_ALIAS_NAME
alb_route53_zone_id = ROUTE53_ZONE_ID

TO create and use HTTPS ALB listener a certificate and DNS zone Id are mandatory. You must provide the following variables:

alb_cert_arn        = ALB_CERT_ARN
alb_alias_name      = ALB_ALIAS_NAME
alb_route53_zone_id = ROUTE53_ZONE_ID

OBS

We support tiering to s3. In order to setup tiering, you must supply the following variables:

tiering_enable_obs_integration = true
tiering_obs_name               = "..."

In addition, you can supply (and override our default):

tiering_ssd_percent = VALUE

Clients

prerequisites:

  • client_instance_iam_profile_arn
Clients iam policy (replace *prefix* and *cluster_name* with relevant values)
{
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
      {
        "Action": [
          "ec2:DescribeNetworkInterfaces",
          "ec2:AttachNetworkInterface",
          "ec2:CreateNetworkInterface",
          "ec2:ModifyNetworkInterfaceAttribute",
          "ec2:DeleteNetworkInterface",
          "ec2:DescribeInstances",
          "ec2:DescribeInstanceTypes"
        ],
        "Effect": "Allow",
        "Resource": "*"
      },
      {
        "Action": [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "logs:DescribeLogStreams",
          "logs:PutRetentionPolicy"
        ],
        "Effect": "Allow",
        "Resource": [
          "arn:aws:logs:*:*:log-group:/wekaio/clients/prefix-cluster_name-client*"
        ]
      }
    ],
    "Version": "2012-10-17"
}

We support creating clients that will be mounted automatically to the cluster.
In order to create clients you need to provide the number of clients you want (by default the number is 0), for example:

clients_number = 2

This will automatically create 2 clients.
In addition you can provide these optional variables:

client_instance_type   = "c5.2xlarge"
client_nics_num        = DESIRED_NUM
client_instance_ami_id = AMI_ID
client_arch            = "x86_64"


In order to use exising iam instance profile ARN you need to provide the following variable:

client_instance_iam_profile_arn = CLIENT_ARN

NFS Protocol Gateways

We support creating protocol gateways that will be mounted automatically to the cluster.
In order to create you need to provide the number of protocol gateways instances you want (by default the number is 0), for example:

nfs_protocol_gateways_number = 2

This will automatically create 2 instances.
In addition you can supply these optional variables:

nfs_protocol_gateway_secondary_ips_per_nic    = 3
nfs_protocol_gateway_instance_type            = "c5.2xlarge"
nfs_protocol_gateway_nics_num                 = 2
nfs_protocol_gateway_disk_size                = 48
nfs_protocol_gateway_fe_cores_num             = 1
nfs_protocol_gateway_instance_iam_profile_arn = ""
nfs_protocol_gateway_instance_ami_id          = "<YOUR AMI_ID>"


In order to create stateless clients, need to set variable:

nfs_setup_protocol = true

prerequisites:

  • protocol_gateway_instance_iam_profile_arn
Protocol gateway iam policy (replace *prefix*, *cluster_name* and *gateways_name* with relevant values)
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action":
    [
      "ec2:DescribeNetworkInterfaces",
      "ec2:AttachNetworkInterface",
      "ec2:CreateNetworkInterface",
      "ec2:ModifyNetworkInterfaceAttribute",
      "ec2:DeleteNetworkInterface",
      "ec2:DescribeInstances",
      "ec2:DescribeTags",
      "ec2:AssignPrivateIpAddresses"
    ],
    "Resource":  "*",
    },
    {
      "Effect": "Allow",
      "Action":
    [
      "secretsmanager:GetSecretValue"
    ]
    "Resource":
    [
      "arn:aws:secretsmanager:*:*:secret:weka/prefix-cluster_name/*"
    ]
    },
    {
      "Effect": "Allow",
      "Action":
    [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
      "logs:DescribeLogStreams",
      "logs:PutRetentionPolicy"
    ],
    "Resource":
    [
      "arn:aws:logs:*:*:log-group:/wekaio/clients/gateways_name*"
    ]
    },
    {
      "Effect": "Allow",
      "Action":
    [
      "autoscaling:DescribeAutoScalingGroups"
    ],
    "Resource":
    [
      "*"
    ]
    },
    {
      "Action": [
        "lambda:InvokeFunction"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:lambda:*:*:function:prefix-cluster_name*"
      ]
    },
  ]
}

S3 Protocol Gateways

We support creating protocol gateways that will be mounted automatically to the cluster.
In order to create you need to provide the number of protocol gateways instances you want (by default the number is 0),

for example:

s3_protocol_gateways_number = 2

This will automatically create 2 instances.
In addition you can supply these optional variables:

s3_protocol_gateway_instance_type            = "c5.2xlarge"
s3_protocol_gateway_disk_size                = 48
s3_protocol_gateway_fe_cores_num             = 1
s3_protocol_gateway_instance_iam_profile_arn = "<YOUR ARN>"
s3_protocol_gateway_instance_ami_id          = "<YOUR AMI_ID>"

SMB Protocol Gateways

We support creating protocol gateways that will be mounted automatically to the cluster.
In order to create you need to provide the number of protocol gateways instances you want (by default the number is 0),

The amount of SMB protocol gateways should be at least 3.
for example:

smb_protocol_gateways_number = 3

This will automatically create 2 instances.
In addition you can supply these optional variables:

smb_protocol_gateway_secondary_ips_per_nic    = 3
smb_protocol_gateway_instance_type            = "c5.2xlarge"
smb_protocol_gateway_nics_num                 = 2
smb_protocol_gateway_disk_size                = 48
smb_protocol_gateway_fe_cores_num             = 1
smb_protocol_gateway_instance_iam_profile_arn = ""
smb_cluster_name                              = ""
smb_domain_name                               = ""
smb_protocol_gateway_instance_ami_id          = "<YOUR AMI_ID>"


In order to create stateless clients, need to set variable:

smb_setup_protocol = true


In order to enable SMBW, need to set variable:

smbw_enabled = true

To join an SMB cluster in Active Directory, need to run manually command:

weka smb domain join <smb_domain_username> <smb_domain_password> [--server smb_server_name].

prerequisites:

  • protocol_gateway_instance_iam_profile_arn
Protocol gateway iam policy (replace *prefix*, *cluster_name* and *gateways_name* with relevant values)
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action":
    [
      "ec2:DescribeNetworkInterfaces",
      "ec2:AttachNetworkInterface",
      "ec2:CreateNetworkInterface",
      "ec2:ModifyNetworkInterfaceAttribute",
      "ec2:DeleteNetworkInterface",
      "ec2:DescribeInstances",
      "ec2:DescribeTags"
    ],
    "Resource":  "*",
    },
    {
      "Effect": "Allow",
      "Action":
    [
      "secretsmanager:GetSecretValue"
    ]
    "Resource":
    [
      "arn:aws:secretsmanager:*:*:secret:weka/prefix-cluster_name/*"
    ]
    },
    {
      "Effect": "Allow",
      "Action":
    [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
      "logs:DescribeLogStreams",
      "logs:PutRetentionPolicy"
    ]
    "Resource":
    [
      "arn:aws:logs:*:*:log-group:/wekaio/clients/gateways_name*"
    ]
    },
    {
      "Effect": "Allow",
      "Action":
    [
      "autoscaling:DescribeAutoScalingGroups"
    ],
    "Resource":
    [
      "*"
    ]
    },
    {
      "Action": [
        "lambda:InvokeFunction"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:lambda:*:*:function:prefix-cluster_name*"
      ]
    }
  ]
}

Secret manager

By default, if not provided explicitly to the module, we will set:

secretmanager_use_vpc_endpoint = true
secretmanager_create_vpc_endpoint = true

This means we will create a secretmanager endpoint and will use it in the scale down lambda function.
If a secretmanager endpoint already exists, then set:

secretmanager_create_vpc_endpoint = false

It is possible to not use the secretmanager endpoint, but not recommended.
To not use the secretmanager endpoint, set:

secretmanager_use_vpc_endpoint = false
secretmanager_create_vpc_endpoint = false

Run lambdas inside vpc:

To enable vpc config for lambdas, set:

enable_lambda_vpc_config = true

If network VPC is not configured with a NAT gateway, the following needs to be set:

vpc_endpoint_ec2_create              = true
vpc_endpoint_lambda_create           = true
vpc_endpoint_dynamodb_gateway_create = true
vpc_endpoint_autoscaling_create      = true

Further explanation:

We use the secret manager to store the weka username, password (and get.weka.io token).
We need to be able to use them on scale down lambda which runs inside the provided vpc.
This is the reason we need the secret manager endpoint on the vpc.
In case setting secret manager endpoint isn't possible, you will need to set the variables as described above.
In this case the weka password will be shown as plain text on the state machine, since it will need to be sent from the fetch lambda to the scale down lambda.

Endpoints

In case you want to deploy a weka cluster inside a vpc with no internet access, you will need to set the following endpoints:

vpc endpoint proxy

We need an endpoint to reach home.weka.io, get.weka.io, and AWS EC2/cloudwatch services.
To use weka vpc endpoint service, set:

vpc_endpoint_proxy_create = true

Alternatively appropriate customer-managed proxy can be provided by proxy_url variable:

proxy_url = "..."

vpc endpoint ec2

Weka deployment requires access to EC2 services.
To let terraform create ec2 endpoint, set:

vpc_endpoint_ec2_create = true

vpc endpoint s3 gateway

Weka deployment requires access to S3 services.
To let terraform create s3 gateway, set:

vpc_endpoint_s3_gateway_create = true

vpc endpoint lambda

Weka deployment requires access to lambda services.
To let terraform create lambda endpoint, set:

vpc_endpoint_s3_gateway_create = true

Terraform output

The module output contains useful information about the created resources. For example: ssh username, weka password secret id etc. The helper_commands part in the output provides lambda call that can be used to learn about the clusterization process.

Requirements

Name Version
terraform >= 1.4.6
aws >= 5.5.0
local >= 2.0.0
random >= 3.5.0
tls >= 4.0.0

Providers

Name Version
aws >= 5.5.0
local >= 2.0.0
random >= 3.5.0
tls >= 4.0.0

Modules

Name Source Version
clients ./modules/clients n/a
iam ./modules/iam n/a
network ./modules/network n/a
nfs_protocol_gateways ./modules/protocol_gateways n/a
s3_protocol_gateways ./modules/protocol_gateways n/a
security_group ./modules/security_group n/a
smb_protocol_gateways ./modules/protocol_gateways n/a
vpc_endpoint ./modules/endpoint n/a

Resources

Name Type
aws_autoscaling_attachment.alb_autoscaling_attachment resource
aws_autoscaling_group.autoscaling_group resource
aws_cloudwatch_event_rule.event_rule resource
aws_cloudwatch_event_target.step_function_event_target resource
aws_cloudwatch_log_group.cloudwatch_log_group resource
aws_cloudwatch_log_group.sfn_log_group resource
aws_dynamodb_table.weka_deployment resource
aws_dynamodb_table_item.weka_deployment_nfs_state resource
aws_dynamodb_table_item.weka_deployment_state resource
aws_key_pair.generated_key resource
aws_kms_alias.kms_alias resource
aws_kms_key.kms_key resource
aws_kms_key_policy.kms_key_policy resource
aws_lambda_function.clusterize_finalization_lambda resource
aws_lambda_function.clusterize_lambda resource
aws_lambda_function.deploy_lambda resource
aws_lambda_function.fetch_lambda resource
aws_lambda_function.join_nfs_finalization_lambda resource
aws_lambda_function.management resource
aws_lambda_function.report_lambda resource
aws_lambda_function.scale_down_lambda resource
aws_lambda_function.status_lambda resource
aws_lambda_function.terminate_lambda resource
aws_lambda_function.transient_lambda resource
aws_lambda_permission.invoke_lambda_permission resource
aws_launch_template.launch_template resource
aws_lb.alb resource
aws_lb_listener.lb_listener resource
aws_lb_listener.lb_weka_listener resource
aws_lb_target_group.alb_target_group resource
aws_placement_group.placement_group resource
aws_route53_record.lb_record resource
aws_secretsmanager_secret.get_weka_io_token resource
aws_secretsmanager_secret.weka_deployment_password resource
aws_secretsmanager_secret.weka_password resource
aws_secretsmanager_secret.weka_username resource
aws_secretsmanager_secret_version.get_weka_io_token resource
aws_secretsmanager_secret_version.weka_username resource
aws_sfn_state_machine.scale_down_state_machine resource
aws_vpc_endpoint.secretmanager_endpoint resource
local_file.private_key resource
local_file.public_key resource
random_password.suffix resource
tls_private_key.key resource
aws_ami.amzn_ami data source
aws_caller_identity.current data source
aws_region.current data source
aws_subnet.this data source

Inputs

Name Description Type Default Required
additional_instance_iam_policy_statement Additional IAM policy statement to be added to the instance IAM role.
list(object({
Effect = string
Action = list(string)
Resource = list(string)
}))
null no
alb_additional_subnet_cidr_block Additional CIDR block for public subnet string "10.0.3.0/24" no
alb_additional_subnet_id Required to specify if subnet_ids were used to specify pre-defined subnets for weka. ALB requires an additional subnet, and in the case of pre-defined networking this one also should be pre-defined string "" no
alb_additional_subnet_zone Zone for the ALB additional subnet string "" no
alb_alias_name Set ALB alias name string "" no
alb_allow_https_cidrs CIDRs to allow connecting to ALB over 443 port, by default 443 is not opened, and port 14000 used for connection, inheriting setting from allow_weka_api_ranges list(string) [] no
alb_cert_arn HTTPS certificate ARN for ALB string null no
alb_route53_zone_id Route53 zone id string "" no
alb_sg_ids Security group ids for ALB list(string) [] no
allow_ssh_cidrs Allow port 22, if not provided, i.e leaving the default empty list, the rule will not be included in the SG list(string) [] no
allow_weka_api_cidrs Allow connection to port 14000 on weka backends and ALB(if exists and not provided with dedicated SG) from specified CIDRs, by default no CIDRs are allowed. All ports (including 14000) are allowed within VPC list(string) [] no
ami_id AMI ID to use, Amazon Linux 2 is the supported OS. string null no
assign_public_ip Determines whether to assign public IP to all instances deployed by TF module. Includes backends, clients and protocol gateways string "auto" no
availability_zones Required only if not specifying subnet_ids, this zone(s) will be used to create subnet that will be used by weka. Currently limited to single subnet list(string) [] no
backends_weka_volume_size The backends' default disk size. number 48 no
capacity_reservation_id The ID of the Capacity Reservation in which to run the backends string null no
client_arch Use arch for ami id, value can be arm64/x86_64. string null no
client_capacity_reservation_id The ID of the capacity reservation in which to run the clients string null no
client_frontend_cores Number of frontend cores to use on client instances, this number will reflect on number of NICs attached to instance, as each weka core requires dedicated NIC number 1 no
client_instance_ami_id The default AMI ID is set to Amazon Linux 2. For the list of all supported Weka Client OSs please refer to: https://docs.weka.io/planning-and-installation/prerequisites-and-compatibility#operating-system string null no
client_instance_iam_profile_arn ARN of IAM Instance Profile to use by client instance. If not specified Instance Profile will be automatically created string "" no
client_instance_type The client instance type (sku) to deploy string "c5.2xlarge" no
client_placement_group_name The client instances placement group name. Backend placement group can be reused. If not specified placement group will be created automatically string null no
client_use_backends_placement_group Use backends placement group for clients bool true no
client_weka_volume_device_name The client volume device name string "/dev/xvda" no
client_weka_volume_size The client volume size in GB number 48 no
clients_custom_data Custom data to pass to the client instances string "" no
clients_number The number of client instances to deploy number 0 no
clients_use_autoscaling_group Use autoscaling group for clients bool false no
clients_use_dpdk Mount weka clients in DPDK mode bool true no
cluster_name The cluster name. string "poc" no
cluster_size The number of virtual machines to deploy. number 6 no
containers_config_map Maps the number of objects and memory size per machine type.
map(object({
compute = number
drive = number
frontend = number
nvme = number
nics = number
memory = list(string)
}))
{
"i3en.12xlarge": {
"compute": 4,
"drive": 2,
"frontend": 1,
"memory": [
"310.7GB",
"310.4GB"
],
"nics": 8,
"nvme": 4
},
"i3en.24xlarge": {
"compute": 9,
"drive": 4,
"frontend": 1,
"memory": [
"384GB",
"384GB"
],
"nics": 15,
"nvme": 8
},
"i3en.2xlarge": {
"compute": 1,
"drive": 1,
"frontend": 1,
"memory": [
"32.9GB",
"32.64GB"
],
"nics": 4,
"nvme": 2
},
"i3en.3xlarge": {
"compute": 1,
"drive": 1,
"frontend": 1,
"memory": [
"62GB",
"61.7GB"
],
"nics": 4,
"nvme": 1
},
"i3en.6xlarge": {
"compute": 4,
"drive": 2,
"frontend": 1,
"memory": [
"136.5GB",
"136.2GB"
],
"nics": 8,
"nvme": 2
}
}
no
create_alb Create ALB for backend UI, and joining weka clients will use this ALB to join a cluster, allowing for better distribution of load amongst backends bool true no
create_nat_gateway NAT needs to be created when no public ip is assigned to the backend, to allow internet access bool false no
custom_data Custom data to pass to instances. string "" no
debug_down_backends_removal_timeout Don't change this value without consulting weka support team. Timeout for removing down backends. Valid time units are ns, us (or µs), ms, s, m, h. string "3h" no
dynamodb_hash_key_name DynamoDB hash key name (optional configuration, will use 'Key' by default). This key will be used if dynamodb table will be created automatically, by not setting dynamodb_table_name param. In case dynamodb_table_name parameter is set, dynamodb_hash_key_name should match the key that should be used by us within pre-created table string "Key" no
dynamodb_table_name DynamoDB table name, if not supplied a new table will be created string "" no
ebs_encrypted Enables EBS encryption on the volume bool true no
ebs_kms_key_id The ARN of the AWS Key Management Service string null no
enable_key_pair create / use key pair for instance template bool true no
enable_lambda_vpc_config Config lambda to run inside vpc bool false no
event_iam_role_arn IAM Role that will be used by cloudwatch rule(event), if not specified will be created automatically. If pre-created should match policy described in readme string "" no
get_weka_io_token The token to download the Weka release from get.weka.io. string n/a yes
get_weka_io_token_secret_id The secrets manager secret id of the token to download the Weka release from get.weka.io. i.e. arn:aws:secretsmanager::<ACCOUNT_NUMBER>:secret:<SECRET_NAME> string "" no
hotspare Number of hotspares to set on weka cluster. Refer to https://docs.weka.io/overview/ssd-capacity-management#hot-spare number 1 no
install_cluster_dpdk Install weka cluster with DPDK bool true no
install_weka_url The URL of the Weka release. Supports path to weka tar file or installation script. string "" no
instance_iam_profile_arn ARN of IAM Instance Profile that will be used by weka backend instances, if not specified will be created automatically. If pre-created should match policy described in readme string "" no
instance_type The virtual machine type (sku) to deploy. string "i3en.2xlarge" no
key_pair_name Ssh key pair name to pass to the instances. string null no
lambda_iam_role_arn IAM Role that will be used by AWS Lambdas, if not specified will be created automatically. If pre-created should match policy described in readme string "" no
lambdas_custom_s3_bucket S3 bucket name for lambdas string null no
lambdas_custom_s3_key S3 key for lambdas string null no
lambdas_dist Lambdas code dist string "release" no
lambdas_version Lambdas code version (hash) string "23b310b040a6ea53217b8d7ef8d1d275" no
metadata_http_tokens Whether or not the metadata service requires session tokens, also referred to as Instance Metadata Service Version 2 (IMDSv2) string "required" no
nat_public_subnet_cidr CIDR block for public subnet string "10.0.2.0/24" no
nfs_capacity_reservation_id The ID of the capacity reservation in which to run the nfs clients string null no
nfs_interface_group_name Interface group name. string "weka-ig" no
nfs_protocol_gateway_fe_cores_num The protocol gateways' NICs number. number 1 no
nfs_protocol_gateway_instance_ami_id AMI ID to use, Amazon Linux 2 is the supported OS. string null no
nfs_protocol_gateway_instance_iam_profile_arn The protocol gateway instance IAM profile ARN string "" no
nfs_protocol_gateway_instance_type The protocol gateways' virtual machine type (sku) to deploy. string "c5n.2xlarge" no
nfs_protocol_gateway_secondary_ips_per_nic Number of secondary IPs per single NIC per protocol gateway virtual machine. number 3 no
nfs_protocol_gateway_weka_volume_size The protocol gateways' default disk size. number 48 no
nfs_protocol_gateways_number The number of protocol gateway virtual machines to deploy. number 0 no
nfs_setup_protocol Setup protocol, default if false bool false no
placement_group_name n/a string null no
prefix Prefix for all resources string "weka" no
protection_level Cluster data protection level. number 2 no
proxy_url Weka proxy url string "" no
s3_capacity_reservation_id The ID of the capacity reservation in which to run the s3 clients string null no
s3_protocol_gateway_fe_cores_num S3 protocol gateways' NICs number. number 1 no
s3_protocol_gateway_instance_ami_id AMI ID to use, Amazon Linux 2 is the supported OS. string null no
s3_protocol_gateway_instance_iam_profile_arn The protocol gateway instance IAM profile ARN string "" no
s3_protocol_gateway_instance_type The protocol gateways' virtual machine type (sku) to deploy. string "c5n.2xlarge" no
s3_protocol_gateway_weka_volume_size The protocol gateways' default disk size. number 48 no
s3_protocol_gateways_number The number of protocol gateway virtual machines to deploy. number 0 no
s3_setup_protocol Config protocol, default if false bool false no
secretmanager_create_vpc_endpoint Enable secret manager VPC endpoint bool true no
secretmanager_sg_ids Secret manager endpoint security groups ids list(string) [] no
secretmanager_use_vpc_endpoint Use of secret manager is optional, if not used secrets will be passed between lambdas over step function. If secret manager is used, all lambdas will fetch secret directly when needed. bool true no
set_dedicated_fe_container Create cluster with FE containers bool true no
sfn_iam_role_arn Step function iam role arn string "" no
sg_ids Security group ids list(string) [] no
smb_capacity_reservation_id The ID of the capacity reservation in which to run the smb clients string null no
smb_cluster_name The name of the SMB setup. string "Weka-SMB" no
smb_domain_name The domain to join the SMB cluster to. string "" no
smb_protocol_gateway_fe_cores_num The protocol gateways' NICs number. number 1 no
smb_protocol_gateway_instance_ami_id AMI ID to use, Amazon Linux 2 is the supported OS. string null no
smb_protocol_gateway_instance_iam_profile_arn The protocol gateway instance IAM profile ARN string "" no
smb_protocol_gateway_instance_type The protocol gateways' virtual machine type (sku) to deploy. string "c5n.2xlarge" no
smb_protocol_gateway_secondary_ips_per_nic Number of secondary IPs per single NIC per protocol gateway virtual machine. number 0 no
smb_protocol_gateway_weka_volume_size The protocol gateways' default disk size. number 48 no
smb_protocol_gateways_number The number of protocol gateway virtual machines to deploy. number 0 no
smb_setup_protocol Config protocol, default if false bool false no
smbw_enabled Enable SMBW protocol. This option should be provided before cluster is created to leave extra capacity for SMBW setup. bool true no
ssh_public_key Ssh public key to pass to the instances. string null no
stripe_width Stripe width = cluster_size - protection_level - 1 (by default). number -1 no
subnet_autocreate_as_private Create private subnet using nat gateway to route traffic. The default is public network. Relevant only when subnet_ids is empty. bool false no
subnet_ids List of subnet ids list(string) [] no
subnets_cidrs CIDR block for subnet creation, required only if not specifying subnet_ids, this block will be used to create subnet that will be used by weka. Currently limited to single list(string)
[
"10.0.1.0/24"
]
no
tags_map A map of tags to assign the same metadata to all resources in the environment. Format: key:value. Note: Manually tagged resources will be overridden by Terraform apply. map(string) {} no
tiering_enable_obs_integration Determines whether to enable object stores integration with the Weka cluster. Set true to enable the integration. bool false no
tiering_enable_ssd_percent When set_obs_integration is true, this variable sets the capacity percentage of the filesystem that resides on SSD. For example, for an SSD with a total capacity of 20GB, and the tiering_ssd_percent is set to 20, the total available capacity is 100GB. number 20 no
tiering_obs_name Name of an existing S3 bucket string "" no
tiering_obs_start_demote Target tiering cue (in seconds) before starting upload data to OBS (turning it into read cache). Default is 10 seconds. number 10 no
tiering_obs_target_ssd_retention Target retention period (in seconds) before tiering to OBS (how long data will stay in SSD). Default is 86400 seconds (24 hours). number 86400 no
use_placement_group Use cluster placement group for backends. Note: If not using a cluster placement group, the instances will most likely be spread out across the underlying AWS infrastructure, resulting in not getting the maximum performance from the WEKA cluster bool true no
vpc_cidr CIDR block of the vpc string "10.0.0.0/16" no
vpc_endpoint_autoscaling_create Create autoscaling VPC endpoint bool false no
vpc_endpoint_dynamodb_gateway_create Create dynamodb gateway VPC endpoint bool false no
vpc_endpoint_ec2_create Create Ec2 VPC endpoint bool false no
vpc_endpoint_lambda_create Create Ec2 VPC endpoint bool false no
vpc_endpoint_proxy_create creates VPC endpoint to weka-provided VPC Endpoint services that enable managed proxy to reach home.weka.io, get.weka.io, and AWS EC2/cloudwatch services”. Alternatively appropriate customer-managed proxy can be provided by proxy_url variable bool false no
vpc_endpoint_s3_gateway_create Create S3 gateway VPC endpoint bool false no
vpc_id VPC ID, required only for security group creation string "" no
weka_home_url Weka Home url string "" no
weka_version The Weka version to deploy. string "" no

Outputs

Name Description
alb_alias_record If 'alb_alias_name not null, it will output fqdn of the ALB
alb_dns_name If 'create_alb set to true, it will output dns name of the ALB
asg_name Name of ASG
client_asg_name n/a
client_helper_commands n/a
client_ips Ips of clients
cluster_helper_commands n/a
cluster_name The cluster name
deploy_lambda_name n/a
ips_type If 'assign_public_ip' is set to true, it will output the public ips, If no it will output the private ips
lambda_status_name Name of lambda status
local_ssh_private_key If 'ssh_public_key' is set to null and no key_pair_name provided, it will output the private ssh key location.
nfs_protocol_gateways_ips Ips of NFS protocol gateways
nfs_protocol_gateways_name Name of NFS protocol gateway instances
placement_group_name Name of placement group
pre_terraform_destroy_command Mandatory pre-destroy steps only when S3/SMB protocol gateways are crated. Terraform doesn't handle protection removal.
s3_protocol_gateways_ips Ips of S3 protocol gateways
s3_protocol_gateways_name Name of S3 protocol gateway instances
sg_ids Security group ids of backends
smb_protocol_gateways_ips Ips of SMB protocol gateways
smb_protocol_gateways_name Name of SMB protocol gateway instances
subnet_ids Subnet ids of backends
vpc_id VPC id
weka_cluster_admin_password_secret_id Secret id of weka admin password