This module handles opinionated Google Cloud Platform Kubernetes Engine cluster creation and configuration with Node Pools, IP MASQ, Network Policy, etc. The resources/services/activations/deletions that this module will create/trigger are:

• Create a GKE cluster with the provided addons
• Create GKE Node Pool(s) with provided configuration and attach to cluster
• Replace the default kube-dns configmap if stub_domains are provided
• Activate network policy if network_policy is true
• Add ip-masq-agent configmap with provided non_masquerade_cidrs if configure_ip_masq is true

Sub modules are provided for creating private clusters, beta private clusters, and beta public clusters as well. Beta sub modules allow for the use of various GKE beta features. See the modules directory for the various sub modules.

This module is meant for use with Terraform 1.3+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=1.3, please open an issue.

If you haven't upgraded to 1.3 and need a Terraform 0.13.x-compatible version of this module, the last released version intended for Terraform 0.13.x is [27.0.0].

If you haven't upgraded to 0.13 and need a Terraform 0.12.x-compatible version of this module, the last released version intended for Terraform 0.12.x is 12.3.0.

There are multiple examples included in the examples folder but simple usage is as follows:

# google_client_config and kubernetes provider must be explicitly specified like the following.
data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${module.gke.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(module.gke.ca_certificate)
}

module "gke" {
  source                     = "terraform-google-modules/kubernetes-engine/google"
  project_id                 = "<PROJECT ID>"
  name                       = "gke-test-1"
  region                     = "us-central1"
  zones                      = ["us-central1-a", "us-central1-b", "us-central1-f"]
  network                    = "vpc-01"
  subnetwork                 = "us-central1-01"
  ip_range_pods              = "us-central1-01-gke-01-pods"
  ip_range_services          = "us-central1-01-gke-01-services"
  http_load_balancing        = false
  network_policy             = false
  horizontal_pod_autoscaling = true
  filestore_csi_driver       = false
  dns_cache                  = false

  node_pools = [
    {
      name                        = "default-node-pool"
      machine_type                = "e2-medium"
      node_locations              = "us-central1-b,us-central1-c"
      min_count                   = 1
      max_count                   = 100
      local_ssd_count             = 0
      spot                        = false
      disk_size_gb                = 100
      disk_type                   = "pd-standard"
      image_type                  = "COS_CONTAINERD"
      enable_gcfs                 = false
      enable_gvnic                = false
      logging_variant             = "DEFAULT"
      auto_repair                 = true
      auto_upgrade                = true
      service_account             = "project-service-account@<PROJECT ID>.iam.gserviceaccount.com"
      preemptible                 = false
      initial_node_count          = 80
      accelerator_count           = 1
      accelerator_type            = "nvidia-l4"
      gpu_driver_version          = "LATEST"
      gpu_sharing_strategy        = "TIME_SHARING"
      max_shared_clients_per_gpu = 2
    },
  ]

  node_pools_oauth_scopes = {
    all = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
  }

  node_pools_labels = {
    all = {}

    default-node-pool = {
      default-node-pool = true
    }
  }

  node_pools_metadata = {
    all = {}

    default-node-pool = {
      node-pool-metadata-custom-value = "my-node-pool"
    }
  }

  node_pools_taints = {
    all = []

    default-node-pool = [
      {
        key    = "default-node-pool"
        value  = true
        effect = "PREFER_NO_SCHEDULE"
      },
    ]
  }

  node_pools_tags = {
    all = []

    default-node-pool = [
      "default-node-pool",
    ]
  }
}

Then perform the following commands on the root folder:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

object({
    enabled                     = bool
    autoscaling_profile         = string
    min_cpu_cores               = number
    max_cpu_cores               = number
    min_memory_gb               = number
    max_memory_gb               = number
    gpu_resources               = list(object({ resource_type = string, minimum = number, maximum = number }))
    auto_repair                 = bool
    auto_upgrade                = bool
    disk_size                   = optional(number)
    disk_type                   = optional(string)
    image_type                  = optional(string)
    strategy                    = optional(string)
    max_surge                   = optional(number)
    max_unavailable             = optional(number)
    node_pool_soak_duration     = optional(string)
    batch_soak_duration         = optional(string)
    batch_percentage            = optional(number)
    batch_node_count            = optional(number)
    enable_secure_boot          = optional(bool, false)
    enable_integrity_monitoring = optional(bool, true)
  })

{
  "auto_repair": true,
  "auto_upgrade": true,
  "autoscaling_profile": "BALANCED",
  "disk_size": 100,
  "disk_type": "pd-standard",
  "enable_integrity_monitoring": true,
  "enable_secure_boot": false,
  "enabled": false,
  "gpu_resources": [],
  "image_type": "COS_CONTAINERD",
  "max_cpu_cores": 0,
  "max_memory_gb": 0,
  "min_cpu_cores": 0,
  "min_memory_gb": 0
}

[
  {
    "key_name": "",
    "state": "DECRYPTED"
  }
]

[
  "8443",
  "9443",
  "15017"
]

[
  {
    "name": "default-node-pool"
  }
]

{
  "all": "",
  "default-node-pool": ""
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": [
    "https://www.googleapis.com/auth/cloud-platform"
  ],
  "default-node-pool": []
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": [],
  "default-node-pool": []
}

{
  "all": [],
  "default-node-pool": []
}

[
  "10.0.0.0/8",
  "172.16.0.0/12",
  "192.168.0.0/16"
]

object({
    enabled            = bool
    logging_enabled    = optional(bool, false)
    monitoring_enabled = optional(bool, false)
  })

{
  "enabled": false,
  "logging_enabled": false,
  "monitoring_enabled": false
}

object({
    metadata = string
  })

{
  "metadata": "INCLUDE_ALL_METADATA"
}

Use this variable for provisioning linux based node pools. For Windows based node pools use windows_node_pools

The node_pools variable takes the following parameters:

The windows_node_pools variable takes the same parameters as node_pools but is reserved for provisioning Windows based node pools only. This variable is introduced to satisfy a specific requirement for the presence of at least one linux based node pool in the cluster before a windows based node pool can be created.

Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:

1. Terraform and kubectl are installed on the machine where Terraform is executed.
2. The Service Account you execute the module with has the right permissions.
3. The Compute Engine and Kubernetes Engine APIs are active on the project you will launch the cluster in.
4. If you are using a Shared VPC, the APIs must also be activated on the Shared VPC host project and your service account needs the proper permissions there.

The project factory can be used to provision projects with the correct APIs active and the necessary Shared VPC connections.

• kubectl 1.9.x

• Terraform 1.3+
• Terraform Provider for GCP v5.9+

Some submodules use the terraform-google-gcloud module. By default, this module assumes you already have gcloud installed in your $PATH. See the module documentation for more information.

In order to execute this module you must have a Service Account with the following project roles:

• roles/compute.viewer
• roles/compute.securityAdmin (only required if add_cluster_firewall_rules is set to true)
• roles/container.clusterAdmin
• roles/container.developer
• roles/iam.serviceAccountAdmin
• roles/iam.serviceAccountUser
• roles/resourcemanager.projectIamAdmin (only required if service_account is set to create)

Additionally, if service_account is set to create and grant_registry_access is requested, the service account requires the following role on the registry_project_ids projects:

• roles/resourcemanager.projectIamAdmin

In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:

• Compute Engine API - compute.googleapis.com
• Kubernetes Engine API - container.googleapis.com