jupyter/terraform_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# README (Ignore if you are running on Mac/Linux)\n",
    "\n",
    "If you are running on Windows, make sure you have started the Jupyter Notebook in a Bash shell.\n",
    "Moreover, all the requirements below must be installed in this Bash (compatible) shell.\n",
    "\n",
    "This can be achieved as follows:\n",
    "\n",
    "1. Enable and install WSL(2) for Windows 10/11 [official documentation](https://docs.microsoft.com/en-us/windows/wsl/install)\n",
    "    * On newer builds of W10/11 you can install WSL by running the following command in an *administrator* PowerShell terminal. Which will install by default an Ubuntu instance of WSL.\n",
    "    ```bash\n",
    "   wsl --install\n",
    "    ```\n",
    "2. Start the Ubuntu Bash shell by searching for `Bash` under Start, or by running `bash` in a (normal) PowerShell terminal.\n",
    "\n",
    "Using a Bash terminal as started under step 2 above, you can install the Requirements as described below as if you are running it under Linux or Ubuntu/Debian."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Requirements\n",
    "These requirements may also be installed on Windows, however, development has only been tested on Linux/macOS.\n",
    "\n",
    "Before we get started, first make sure to install all the required tools. We provide two lists below, one needed for setting up the testbed. And one for developing code to use with the testbed. Feel free to skip the installation of the second list, and return at a later point in time.\n",
    "\n",
    "\n",
    "### Deployment\n",
    "\n",
    " > ⚠️ All dependencies must be installed in a Bash-compatible shell. For Windows users also see [above](#read-me)\n",
    "Make sure to install a recent version of each of the dependencies.\n",
    "\n",
    "\n",
    " * (Windows only) Install every dependency in a Windows Subsystem for the Linux, Bash shell (see also README above).\n",
    " * GCloud SDK\n",
    "    - Follow the installation instructions [here](https://cloud.google.com/sdk/docs/install), follow either the Linux installation instruction, or your OS/Distribution specific instructions.\n",
    "    - Initialize the SDK with `gcloud init`, if prompted you may ignore to set/create a default/first project.\n",
    "    - ⚠️ Run the command `gcloud auth application-default login`\n",
    "        - ℹ️ We need to run this command to utilize your login credentials programmatically with terraform. This is needed as we will use these to impersonate a service account during the creation and setup of the Kubernetes cluster.\n",
    "    - ⚠️ Run the command `gcloud components install beta`\n",
    "        - ℹ️ We need to run this command to list the billing account IDs and enable billing. Currently, these features fall under beta access.\n",
    "    - ⚠️ Run the command `gcloud components install gke-gcloud-auth-plugin`\n",
    "        - ℹ️ We need to run this command to retrieve cluster configurations (to be used by `kubectl` and `helm`)\n",
    "    - ⚠️ Run the command `gcloud auth configure-docker`\n",
    "        - ℹ️ We need to run this command to push container images with docker to your project's container registry\n",
    " * Kubectl (>= 1.22.0)\n",
    " * Helm (>= 3.9.4)\n",
    " * Terraform (>= 1.2.8)\n",
    " * Python3.9/10\n",
    "   * jupyter, ipython, bash_kernel\n",
    "```bash\n",
    "pip3 install -r requirements-jupyter.txt\n",
    "python3 -m bash_kernel.install\n",
    "```\n",
    "\n",
    "### Development\n",
    "For development, the following tools are needed/recommended:\n",
    "\n",
    " * Docker (>= 18.09).\n",
    "    - If you don't have experience with using Docker, we recommend following [this](https://docs.docker.com/get-started/) tutorial.\n",
    " * Python3.9\n",
    " * pip3\n",
    " * JetBrains PyCharm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Preparation\n",
    "\n",
    "To make sure we can request resources on Google Cloud Platform (GCP), perform the following;\n",
    "\n",
    "1. Create a GCP account on [https://cloud.google.com](https://cloud.google.com), using a Google account\n",
    "2. Redeem your academic coupon on GCP, see Brightspace for information on obtaining the \\\\$50 academic coupon, or use the free \\\\$300 credits for new users provided by Google.\n",
    "\n",
    "\n",
    "3. Make sure to use the `Bash` kernel, not a Python or other kernel. For those on windows machines, make sure to launch the `jupyter notebook` server from a bash-compliant command line, we recommend Windows Subsystem for Linux.\n",
    "\n",
    "⚠️ Make sure to run this Notebook within a cloned repository, not standalone/downloaded from GitHub.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Deployment\n",
    "\n",
    "⚠️ This notebook assumes that commands are executed in order. Executing the provided commands multiple times should not result in issues. However, re-running cells with `cd` commands, or altering cells (other than variables as instructed) may result in unexpected behaviour.\n",
    "\n",
    "## Getting started\n",
    "\n",
    "First, we will set a few variables used **throughout** the project. We set them in this notebook for convenience, but they are also set to some example default values in configuration files for the project. If you change any of these, make sure to change the corresponding variables as well in;\n",
    "\n",
    "* [`../terraform/terraform-gke/variables.tf`](../terraform/terraform-gke/variables.tf)\n",
    "* [`../terraform/terraform-dependencies/variables.tf`](../terraform/terraform-dependencies/variables.tf)\n",
    "\n",
    "\n",
    "> ⚠️ As you have changed the `PROJECT_ID` parameter to a unique project name, also change the `project_id` variable in the following files. This allows you to run `terraform apply` without having to override the default value for the project.\n",
    "\n",
    "> ℹ️ Any variable changed here can also be provided to `terraform` using the `-var` flag, i.e.  `-var terraform_variable=$BASH_VARIABLE`. An example for setting the `project_id` variable is also provided later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# VARIABLES THAT NEEDS TO BE SET\n",
    "\n",
    "##################\n",
    "### CHANGE ME! ###\n",
    "##################\n",
    "PROJECT_ID=\"test-bed-fltk\"\n",
    "\n",
    "# DEFAULT VARIABLES\n",
    "ACCOUNT_ID=\"terraform-iam-service-account\"\n",
    "PRIVILEGED_ACCOUNT_ID=\"${ACCOUNT_ID}@${PROJECT_ID}.iam.gserviceaccount.com\"\n",
    "CLUSTER_NAME=\"fltk-testbed-cluster\"\n",
    "DEFAULT_POOL=\"default-node-pool\"\n",
    "EXPERIMENT_POOL=\"medium-fltk-pool-1\"\n",
    "REGION=\"us-central1-c\"\n",
    "\n",
    "TERRAFORM_GKE_DIR=\"../terraform/terraform-gke\"\n",
    "TERRAFORM_DEPENDENCIES_DIR=\"../terraform/terraform-dependencies\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Project creation\n",
    "\n",
    "Next, we create a project using the `PROJECT_ID` variable and get all the billing account information.\n",
    "\n",
    "⁉️ (Ignore if using a pre-existing GCP Project) If the command below does not complete successfully, make sure to change the `PROJECT_ID` variable in the previous cell and re-run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gcloud projects create $PROJECT_ID --set-as-default\n",
    "gcloud beta billing accounts list # Copy the Account ID of the account"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Copy the billing account identifier, e.g. `015594-41687F-092941`, and assign to the variable in the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "##################\n",
    "### CHANGE ME! ###\n",
    "##################\n",
    "BILLING_ACCOUNT=\"015594-41687F-092941\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Setup billing and enable services, this will allow us to create a GKE cluster (Google managed Kubernetes cluster), and push and pull containers to our private container repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Setup billing to project\n",
    "gcloud beta billing projects link $PROJECT_ID --billing-account $BILLING_ACCOUNT\n",
    "# Enable services now billing is enabled\n",
    "gcloud services enable compute container --project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Creating a service-account\n",
    "\n",
    "Create service account that has the minimum set of permissions for creating and managing a cluster. This service account\n",
    "will be used to create the cluster, and deploy the dependencies that we use.\n",
    "\n",
    "During the deployment we will make use of impersonation, to let *your* account utilize the service-account. For more information about this practise, see also [this](https://cloud.google.com/blog/topics/developers-practitioners/using-google-cloud-service-account-impersonation-your-terraform-code) blog by Google."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Helper function to quickly enable gcp roles, assumes $PRIVILEGED_ACCOUNT_ID and $PROJECT_ID to be set.\n",
    "function enable_gcp_role () {\n",
    "  ROLE=$1\n",
    "  gcloud projects add-iam-policy-binding \\\n",
    "    $PROJECT_ID \\\n",
    "    --member=\"serviceAccount:$PRIVILEGED_ACCOUNT_ID\" \\\n",
    "    --role=\"roles/$ROLE\"\n",
    "}\n",
    "\n",
    "# Create service-account\n",
    "gcloud iam service-accounts create $ACCOUNT_ID --display-name=\"Terraform service account\" --project ${PROJECT_ID}\n",
    "\n",
    "# Allow the service account to use the the set of roles below.\n",
    "enable_gcp_role \"compute.viewer\"                # Allow the service account to see active resources\n",
    "enable_gcp_role \"storage.objectViewer\"          # Allow the service account/managed resources to pull from gcr.io (your code)\n",
    "enable_gcp_role \"compute.networkAdmin\"          # Needed for setting up private network\n",
    "enable_gcp_role \"compute.securityAdmin\"         # Needed for GKE\n",
    "enable_gcp_role \"container.clusterViewer\"       # Needed for GKE\n",
    "enable_gcp_role \"container.clusterAdmin\"        # Needed for GKE\n",
    "enable_gcp_role \"container.developer\"           # Needed for GKE\n",
    "enable_gcp_role \"iam.serviceAccountAdmin\"       # Needed for GKE\n",
    "enable_gcp_role \"iam.serviceAccountUser\"        # Needed for GKE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Enable impersonation\n",
    "With the service account created, we must enable impersonation, to allow the main account of the project to make use of the service account. For more information see also the [`add-iam-policy-binding`](https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/add-iam-policy-binding) reference.\n",
    "\n",
    "Assign your `google_account` mail to the `OWNER_MAIL` variable, and run the command box below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "##################\n",
    "### CHANGE ME! ###\n",
    "##################\n",
    "OWNER_MAIL=\"mygoogleaccount@gmail.com\"\n",
    "\n",
    "gcloud iam service-accounts add-iam-policy-binding $PRIVILEGED_ACCOUNT_ID \\\n",
    " --member=\"user:$OWNER_MAIL\" \\\n",
    " --role=roles/iam.serviceAccountTokenCreator \\\n",
    " --project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To enable using your account's credentials, run the command below. This will open in a new tab/open the link that is displayed. Afterwards you can use your own credentials to impersonate the service account. \n",
    "\n",
    "You can, for example, also allow other google users (such as project members) to work with your cluster in this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gcloud auth application-default login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Creating a Google managed cluster (GKE)\n",
    "To create the cluster, first change the active directory to the `terraform-gke` directory.\n",
    "\n",
    "⚠️ Creating a cluster will incur billing cost on your project, by default the cluster will be small to minimize costs during this tutorial. Forgetting to `destroy` or scale down the cluster may result in quickly spending your academic coupon.\n",
    "\n",
    "Init the directory, to initialize the Terraform module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_GKE_DIR init "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next, we can check whether we can create a cluster. No warnings or errors should occur during this process. It may take a while to complete.\n",
    "\n",
    "> ⚠️ We provide the project_id variable from `terraform/terraform-gke` manually, and also change the default value.\n",
    "\n",
    "⁉️ If the command below does not complete successfully, e.g. after raising a `403` error, make sure that you have successfully created the project with `gcloud` earlier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_GKE_DIR plan -var project_id=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "When the previous command completes successfully, we can start the deployment. Depending on any changes you may have done, this might take a while.\n",
    "\n",
    "By default, this will create a private zonal cluster consisting of two node pools.\n",
    "\n",
    "> ⚠️ A regional cluster (multi-zonal) will incur an additional fee of \\\\$ 0.10 /hour per managed (GKE) cluster. The **first** zonal cluster is free of this charge.\n",
    "\n",
    "> ⚠️ By default spot/preemptive nodes are disabled. You can experiment by setting `spot` to true in the `tf` files. Note, however, that the default implementations provided in the testbed do not allow for recovery from getting spun down and rescheduled. Moreover, this may result in poor availability during busy hours in the region in which you deploy your cluster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_GKE_DIR apply -auto-approve -var project_id=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next, we add cluster credentials (so you can interact with the cluster through `kubectl` an `helm`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Add credentials for interacting with cluster via kubectl\n",
    "gcloud container clusters get-credentials $CLUSTER_NAME --region $REGION --project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "⚠️ The cluster by default does not contain any nodes in the node pools, the `initial_node_count` is set to 0.\n",
    "\n",
    "Lastly, we need to scale up the cluster, as by default we create a cluster with nodepools of size 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "###\n",
    "### ! CHANGE ME\n",
    "###\n",
    "MAX_NUM_NODES=2\n",
    "\n",
    "gcloud container clusters update $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n",
    "    --no-enable-autoscaling --region $REGION --quiet\n",
    "    \n",
    "# The high performance node will scale up automatically whenever the workloads are deployed\n",
    "gcloud container clusters update $CLUSTER_NAME --node-pool $EXPERIMENT_POOL \\\n",
    "    --enable-autoscaling --min-nodes=0 --max-nodes=$MAX_NUM_NODES --region $REGION --quiet\n",
    "\n",
    "gcloud container clusters resize $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n",
    "    --num-nodes 1 --region $REGION --quiet\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Changing deployment\n",
    "\n",
    "To save cost, or run different experiments, you might want to change the configuration of your cluster. This can be achieved by modifying the cluster configuration in the [`terraform-gke/main.tf`](../terraform/terraform-gke/main.tf) configuration file. You can change the default node-pools, create additional node pools with taints (to allow for scheduling on specific nodes/pools) and much more.\n",
    "\n",
    "After finishing your changes, simply run the following commands\n",
    "\n",
    "```bash\n",
    "# Use `plan` to check your configuration\n",
    "terraform plan\n",
    "# Check to see if your changes are as expected, terraform will show what will be created/removed.\n",
    "\n",
    "# If the changes are as you expect, apply the changes.\n",
    "terraform apply #-auto-approve\n",
    "```\n",
    "\n",
    "Depending on the number of changes, this may take some time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Installing dependencies\n",
    "Lastly, we need to install the dependencies on our cluster. First change the directories, and then run the `init`, `plan` and `apply` commands as we did for creating the GKE cluster.\n",
    "\n",
    "Init the directory, to initialize the Terraform module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR init -reconfigure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Check to see if we can plan the deployment. This will setup the following:\n",
    "\n",
    "* Kubeflow training operator (used to deploy and manage PyTorchTrainJobs programmatically)\n",
    "* NFS-provisioner (used to enable logging on a persistent `ReadWriteMany` PVC in the cluster)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR plan -var project_id=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "When the previous command completes successfully, we can start the deployment. This will install the NFS provisioner and Kubeflow Training Operator dependencies\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR apply -auto-approve -var project_id=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Deploying extractor\n",
    "\n",
    "Lastly, we deploy the extractor pod, which also provides PVCs which can be used for artifact retrieval.\n",
    "\n",
    "Retrieval can be done by running\n",
    "\n",
    "```bash\n",
    "EXTRACTOR_POD_NAME=$(kubectl get pods -n test -l \"app.kubernetes.io/name=fltk.extractor\" -o jsonpath=\"{.items[0].metadata.name}\")\n",
    "kubectl cp -n test $EXTRACTOR_POD_NAME:/opt/federation-lab/logging ./logging\n",
    "```\n",
    "\n",
    "For copying from the extractor path `/opt/federation-lab/logging` to a directory locally named `logging`.\n",
    "\n",
    "First build the docker container, following the instructions of the [readme](https://github.com/JMGaljaard/fltk-testbed#creating-and-uploading-docker-container).\n",
    "\n",
    "\n",
    "N.B. Make sure to have setup a working authentication provider for docker, such that you can push to your repository."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Run this in a terminal in the content-root directory (so `fltk-testbed` if the project name was not altered).\n",
    "```bash\n",
    "python3 -m venv venv\n",
    "source venv\n",
    "pip3 install -r requirements-cpu.txt\n",
    "python3 -m fltk extractor configs/example_cloud_experiment.json\n",
    "```\n",
    "\n",
    "Make sure to have run `gcloud auth configure-docker` in an external terminal.\n",
    "\n",
    "Make sure to allow docker to build/push/run without `sudo` [link](https://cloud.google.com/artifact-registry/docs/docker/authentication).\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Build the docker container with buildkit. Make sure you have Docker Desktop running on Windows/MacOS\n",
    "DOCKER_BUILDKIT=1 docker build --platform linux/amd64 ../ --tag gcr.io/$PROJECT_ID/fltk\n",
    "docker push gcr.io/<project-id>/fltk"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Install the extractor, and set the projectName to $PROJECT_ID.\n",
    "# In case you get a warning regarding the namespace test, this means that the dependencies have not been properly installed.\n",
    "# Make sure to check whether you have enough resources available, and re-run the installation of dependencies. (see above).\n",
    "\n",
    "# Deploy extractor, in test namespace with updated image reference (--set overwrites values from `fltk-values.yaml`).\n",
    "helm install extractor ../charts/extractor -f ../charts/fltk-values.yaml --namespace test --set provider.projectName=$PROJECT_ID"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Testing the deployment\n",
    "\n",
    "To make sure that the deployment went OK, we can run the following command to test whether we can use Pytorch-Training operators.\n",
    "\n",
    "This will create a simple deployment using a Kubeflow pytorch example job.\n",
    "\n",
    "This will create a small (1 master, 1 client) training job on mnist on your cluster. You can follow the deployment by navigating to your cluster on [cloud.google.com](cloud.google.com)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# This cell is optional, but the next shell should show that a pytorch train job is created.\n",
    "kubectl create -f https://raw.githubusercontent.com/kubeflow/training-operator/master/examples/pytorch/simple.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Retrieve all CRD Pytorchjob from Kubeflow.\n",
    "kubectl get pytorchjobs.kubeflow.org --all-namespaces\n",
    "\n",
    "# Alternatively, we can remove all jobs, this will remove all information and logs as well.\n",
    "kubectl delete pytorchjobs.kubeflow.org --all-namespaces --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Cleaning up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling down the cluster\n",
    "\n",
    "This is the preferred way to scale down.\n",
    "Scale node pools down to prevent idle resource utilization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gcloud container clusters resize $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n",
    "     --num-nodes 0 --region $REGION --quiet\n",
    "\n",
    "gcloud container clusters resize $CLUSTER_NAME --node-pool $EXPERIMENT_POOL \\\n",
    "    --num-nodes 0 --region $REGION --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Destroying the cluster\n",
    "\n",
    "> ⚠️ THIS WILL REMOVE YOUR CLUSTER AND DATA STORED ON IT. For this tutorial's purpose destroying your cluster is not an issue. For testing/developing, we recommend manually scaling your cluster up and down instead.\n",
    "\n",
    "To clean up/remove the cluster, we will use the `terraform destroy` command.\n",
    "\n",
    " * Running it in `terraform-dependencies` WILL REMOVE the Kubeflow Training-Operator from your cluster.\n",
    " * Running it in `terraform-gke` WILL REMOVE YOU ENTIRE CLUSTER.\n",
    "\n",
    "You can uncomment the commands below to remove the cluster, or run the command in a terminal in the [`../terraform/terraform-gke`](../terraform/terraform-gke) directory.\n",
    "\n",
    "> ⚠️ It is recommended to scale down the cluster/nodepools rather then destroying, refer to the last code block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# THIS WILL REMOVE/TEARDOWN YOUR CLUSTER, ONLY RECOMMENDED FOR TESTING THE DEPLOYMENT\n",
    "\n",
    "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR destroy -auto-approve -var project_id=$PROJECT_ID\n",
    "\n",
    "terraform -chdir=$TERRAFORM_GKE_DIR destroy -auto-approve -var project_id=$PROJECT_ID"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "title": "Tutorial (Terraform)"
 },
 "nbformat": 4,
 "nbformat_minor": 1
}