-
Notifications
You must be signed in to change notification settings - Fork 845
Bootstrapping GKE cluster for k8s tests
The tests under topgun/k8s
depend on having at least two nodes with different labels, indicating different node images:
- a worker with
label=nodeImage=ubuntu
, and - a worker with
label=nodeImage=cos
.
This is necessary for testing baggageclaim drivers under different node images.
At the time of writing, the default version of GKE, 1.9.7, has an issue with the helm chart. It seems that the YAML rendered by helm from the templates is too long -- there is some hardcoded limit on the length of a configmap (cf. https://github.com/helm/helm/issues/1413).
We then retrieved the correct kubeconfig from GKE, using a command like
gcloud container clusters get-credentials cluster-1 --zone us-central1-a --project cf-concourse-production
We found that this problem could be overcome by deploying using GKE 1.10.7.
Then there were a few steps to follow which have to do with creating an appropriately-authorized service account for the tiller (server-side component of Helm):
# create serviceaccount and clusterrolebinding
kubectl -n kube-system create sa concourse
kubectl -n kube-system create clusterrolebinding concourse --clusterrole=cluster-admin --serviceaccount=kube-system:concourse
token=$(kubectl -n kube-system get secret $(kubectl -n kube-system get sa concourse -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 -D)
kubectl config set-credentials concourse --token "$token"
kubectl config set-context concourse --user concourse --cluster <your-cluster-name>
kubectl config use-context concourse
# create tiller with appropriate permissions
helm init --service-account concourse
where <your-cluster-name>
is the name of the GKE cluster (you can see it in the kubeconfig, or in the output of commands like kubectl config get-contexts
), something like gke_cf-concourse-production_us-central1-a_cluster-1
in our case.
then you can simply copy the contents of the kubeconfig (or at least the sections relevant to your new cluster) into vault, at the path concourse/main/main/kube_config
at the time of writing.