Mannual deploying HA k8s cluster. Base configes and process.
Current supported k8s version: v1.11.6
- k8s suite
- kube-apiserver-amd64:v1.11.6
- kube-controller-manager-amd64:v1.11.6
- kube-scheduler-amd64:v1.11.6
- kube-proxy-amd64:v1.11.6
- etcd-amd64:3.2.18
- pause-amd64:3.1
- k8s-dns-sidecar-amd64:1.14.10 (removed)
- k8s-dns-kube-dns-amd64:1.14.10 (removed)
- k8s-dns-dnsmasq-nanny-amd64:1.14.10 (removed)
- kubernetes-dashboard-amd64:v1.10.1
- cluster-autoscaler:v1.3.3
- network
- pdns
- lvs + keepalived
- coredns:1.5.2 (replacing k8s-dns)
- flannel/traefik
- ingress-nginx:0.25.0
- metallb
- external-dns
- storage
- nfs-client
- postgresql
- mariadb/mysql
- security
- OpenLDAP
- cert-manager
- kubeseal
- keycloak
- metrics
- heapster
- metrics-server
- fluentd-es
- prometheus
- grafana
- efk stack
- service mesh
- istio:v1.2.2
- conduit:v0.5.0 (removed)
- devops infra
- habor + dragonfly
- nexus
- jenkins/gocd/drone
- gogs/gitea/drone
- helm + draft
- kubeapps
- chartmuseum
- sonarqube
- spinnaker
- service catalog
- svc-cat
- automation-broker
- other tools
- netflix suite
Always, it's a good idea that make a whole planning before doing something.
- Prepare server's os, here the oses based on Ubuntu 16.04 LTS. And define server's IP, DNS, other os configs could be modified by shells as later setps.
- Setup a http-server to locate files such as
config
,certs
,docker images
,debs
and some bins likecfssl
etc. - Pre download debs, kube-* bins, docker images, configs, tar/modify them and upload them to the http-server. Pre make certs and upload them too.
- Define some
HTTP_SERVER=<HTTP_SERVER> # the http-server's IP address or FQDN
KUBE_VIP=<KUBE_VIP> # the vip address of k8s masters
- Setup environment for master/nodes.
curl -L https://$HTTP_SERVER/shell/auto-deploy.sh | bash -s env
This step will install needed tools like ssh, curl, ntp, docker runtime, kubelet/kubeadm/kubectl, and upgrade the os to newest stable. Also modify the server's configs like swap, network protocal, ntp configs. Check if all correct.
- Deploy external etcd cluster Suggest deploy etcd cluster on the same master servers. On all masters:
curl -L https://$HTTP_SERVER/shell/auto-deploy.sh | bash -s etcd \
--api-advertise-addresses=$KUBE_VIP \
--etcd-endpoints=https://${ETCD_NODES[0]}:2379,https://${ETCD_NODES[1]}:2379,https://${ETCD_NODES[2]}:2379
Wait etcd service active on all masters, etcd cluster could be ready.
- Deploy the first master On the first master:
curl -L https://$HTTP_SERVER/shell/auto-deploy.sh | bash -s master \
--api-advertise-addresses=$KUBE_VIP \
--etcd-endpoints=https://${ETCD_NODES[0]}:2379,https://${ETCD_NODES[1]}:2379,https://${ETCD_NODES[2]}:2379
When ready, record the token
sha256
from deploy log.
- Deploy other masters On other masters:
curl -L https://$HTTP_SERVER/shell/auto-deploy.sh | bash -s reploca \
--api-advertise-addresses=$KUBE_VIP \
--etcd-endpoints=https://${ETCD_NODES[0]}:2379,https://${ETCD_NODES[1]}:2379,https://${ETCD_NODES[2]}:2379
When all masters ready, the HA is okay.
- Deploy nodes On all nodes:
curl -L https://$HTTP_SERVER/shell/auto-deploy.sh | bash -s node \
--api-advertise-addresses=$KUBE_VIP \
--token <token> --discovery-token-ca-cert-hash sha256:<sha256>
Replace the token
sha256
recorded at setp 3.
- Check cluster
- view k8s cluster info
kubectl cluster-info
- check k8s pods, services, etc.
kubectl get pods,svc --all-namespaces
- check k8s netwoking
route -n
-
Setup netwoks
-
Setup storage
-
Setup secrurity
-
Setup metrics
-
Setup devops infra
-
Setup service mesh
-
Setup other tools
It's a better solution if using Ansible playbook to reduce manual operation.
And it would be the goal that building the deploy process as a service.