This repo will build an AKS or EKS cluster and bootstrap it with the below configuration :

• Dev Dev cluster is deployed in a single region with the enviroments, Dev, QA, Staging being split via K8S namespaces. The application URL is exposed via FrontDoor service which is globally redundant. The backend member is application gateway that sends the traffic to the Nginx ingress controller.
• Prod Prod cluster is deployed either in sigle region or across 2 regions, depending on how the script is run. The application URL is exposed via FrontDoor service which is globally redundant. The backend member is application gateway that sends the traffic to the Nginx ingress controller. The regions are called as PRD-A and PRD-B, and are in active passive mode. If traffic needs to be routed from the secondary region, make changes to the FrontDoor application using the scripts/workflow.

  AKS ingress flow

  Nginx as ingress

  

  Application Gateway(AGIC) as ingress

  

• DefectDojo and SonarQube uses dynamic volumes and are regional resources. The data in these apps are not relicated like a Globally redundant storage account. The Git workflow ensures to update the apps running on both regios and there by keeps them in sync.
• For applications data dont use Dynamic volumes rather use Azure resources like Storage account with GRS, Azure SQL, KeyVault etc that have native globally redundant options available.

• RBAC enabled cluster with 3 nodepools. One is mandatory systempool to run pods in kube-system namespace. The other 2 are optional, edit the Terraform files to update the desired size and number of max/min nodes. The pools are also labelled, so pods will be placed based on node selectors or affinity.

• Creates the following resources in k8s
  • NS: dev, qa, devops-addons, ingress and monitoring.
  • svcaccounts: build-robot account with automountServiceAccountToken: false
  • rbac:
    • k8s role to bind Azure AD group aks-dev-admins to have admin rights on dev namespace.
    • k8s role to bind Azure AD group aks-qa-admins to have admin rights on qa namespace.
    • cluster wide role to grant Azure AD group aks-dev-clus-admins to have admin rights on the Dev cluster.

• If you choose to use Azure Log analytics, run the script with the parameter K8SLogMonitoringType and K8SMetricsMonitoringType set to OMS. For example, .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment DEV -K8SLogMonitoringType OMS -K8SMetricsMonitoringType OMS

• If you choose to use EFK stack, use the below parameters. .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment DEV -K8SLogMonitoringType EFKinCluster

• If you choose to use Azure Log analytics, run the script with the parameter K8SLogMonitoringType and K8SMetricsMonitoringType set to OMS. For example, .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment DEV -K8SLogMonitoringType OMS -K8SMetricsMonitoringType OMS

• If you choose to use Promtheus in Thanos configuration, use the below parameters. .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment DEV -K8SMetricsMonitoringType HAPrometheus-Thanos

• If you chose Azure log analytics then the control plane logs can be queried using Azure monitor to alert for sensitve operations like K8S secrets read, creating/deleting deployments etc. Review the azmonitor.tf to add more queries as per your needs. Note that Log analytics can be expensive, and for a cluster size with 100 nodes max size during business hours, Log analytics cost per month would be AUD 30 per month = 3000 AUD per month For Azure deployment follow the instructions in https://github.com/g0pinath/k8s-core-infra/blob/master/az-aks/readme.md

• Azure security policies are enabled if you chose OMS as the monitoring type. Note that the K8S Pod Security Policies and Azure Policies are not compatible.

• DefectDojo is used to host the reports from kube-bench and anchore.
  • kube-bench - This validates the cluster against CIS benchmarks on a daily basis. The pod logs are read by a cronjob running on git, and is sent to DefectDojo as a post. anchore - This validates the container images in the cluster and reports any vulnerability. The reports are sent to DefectDojo.
  • SonarQube - this is optional and is used for repos whose source code can be compiled directly. SonarQube cant be used to scan Docker images or code base. To enable DefectDojo and SonarQube. .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment DEV -requireDefectDojo true requireSonarQube true
  • OPA Gatekeeper policies - applies 10 baseline policies, review https://github.com/g0pinath/k8s-core-infra/blob/master/devsecops/k8s-templates/opa-gatekeeper/Readme.md for more details

Refer https://github.com/g0pinath/k8s-core-infra/blob/develop/devsecops/readme.md for more details on this.

• Kured is used for rebooting the worker nodes after patching. Review kured-1.4.5.yml and make changes as required. Default values for reboot is Mon-Sun between 11 pm and 5 am AEST.

For a detailed implementation steps, refer the below link.

https://github.com/g0pinath/k8s-core-infra/blob/develop/az-aks/readme.md

Run the below to create the required SPNs and check them into keyvault. This needs to be run from local PC while logged into Azure with contributor access level on the subscription.

    DEV:
    - .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment DEV -dryRunforGithubActions "true"
    PRD:
    - .\k8s-core-resourcesv2.ps1 -createSPNifNotExists4K8S "true" -k8sEnvironment PRD -dryRunforGithubActions "true"

For a list of secrets to be created refer https://github.com/g0pinath/k8s-core-infra/blob/develop/GitSecretsList.md

Name	Trigger	Status
Release Drafter	When a Git release is published.
AzDev-Kube-Bench-Kube-hunter-reports	When a PR is approved into main branch.
AzDev-Core-Infra-Deployment	on commit to Develop branch.
Slack Notifications	on Git events PR, deployment, commit, release etc.
LintCodeBase	on a commit on any branch
create-release-package	publish the package for every release

Set the below environment variables

 - openssl pkcs12 -in stardotcloudkubexyz.pfx  -out stardotcloudkubexyz.enc.key  # extract private key from PFX
 - openssl rsa -in stardotcloudkubexyz.enc.key -outform PEM -out stardotcloudkubexyz.key # convert the encrypted key to unencrypted key.
 - $base64string = [Convert]::ToBase64String([IO.File]::ReadAllBytes("$pwd/certs/stardotcloudkubexyz.key")) 
 - The value of $base64string should be set for $env:TLS_PRIVATE_KEY or Git secret TLS_PRIVATE_KEY

 - client ID for the SPN defectdojo-nonprod-devsecops

 - tenant ID for metricon in Azure.

 - client secret for the SPN defectdojo-nonprod-devsecops

 - initial admin password to login to DefectDojo. You can change this after logging in.

 - we are storing prometheus data in Azure storage account for LTR. We are currently using sametdevopsnp01 account's key for this var. This can also be set to use S3 instead of Azure storage.

 - if you need SMTP email alerts if something is wrong in K8S cluster, and if the SMTP server needs authentication. 

 - initial password for sonarqube, if none set defaults to admin

The following table lists the configurable parameters of the nginx-ingress chart and their default values.