WindTurbineDetector.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "WindTurbineDetector.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qRSPkQ_ryE9t",
        "colab_type": "text"
      },
      "source": [
        "# Wind Turbine Object Detection from Aerial Imagery Using TensorFlow Object Detection API and Google Colab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vLSXaxZ2aCT",
        "colab_type": "text"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "This notebook provides the full pipeline to perform training and inference for a wind turbine object detection model using publically available aerial images and the [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection). It is designed to run in [Google Colab](https://colab.research.google.com/notebooks/welcome.ipynb), a Jupyter notebook environment running on a virtual machine (VM) that provides free access to a Tesla K80 GPU for up to 12 hours.\n",
        "\n",
        "\n",
        "The aerial image data set used in this notebook is obtained from the [National Agriculture Imagery Program (NAIP) database](https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/) using [USGS EarthExplorer](https://earthexplorer.usgs.gov/). The particular NAIP images used to train, test, and validate this model are from three wind farms located in west-central Iowa containing turbines of varying capacity, style, and manufacturer. A sample NAIP image is presented below in the \"Sample NAIP image\" section. The original NAIP images are 5978 x 7648 so they had to be chopped into smaller individual images to avoid excessive memory use. In addition, the ratio of object size to image size is improved by this operation. An image size of 300 x 300 was chosen since the TensorFlow object detection SSD-based models rescale all input images to this size. \n",
        "\n",
        "A total of 488 images, all containing at least one full wind turbine, were collected and split into train (\\~80%), test (\\~16%), and validate (\\~4%) sets. [LabelImg](https://github.com/tzutalin/labelImg) was then used to label all the images in the train and test sets. Samples of the chopped and annotated images are shown below in the \"Sample chopped and annotated NAIP images\" section. Annotating the images in LabelImg creates an XML file corresponding to each image. These XML files must be converted to CSV and then TFRecords. Sample code for this can be found [here](https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9) or [here](https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html) (among other places)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TahAg7fTspdN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# clone wind-turbine-detector repo\n",
        "!git clone https://github.com/lbborkowski/wind-turbine-detector.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NIL8IS9R57U-",
        "colab_type": "text"
      },
      "source": [
        "### Sample NAIP image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wARV6z136BrQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from matplotlib import pyplot as plt\n",
        "from PIL import Image\n",
        "import os\n",
        "import glob\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "image = Image.open('/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg')\n",
        "plt.figure(figsize=(12,8))\n",
        "plt.axis('off')\n",
        "plt.imshow(image)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xjj9v7yf8He4",
        "colab_type": "text"
      },
      "source": [
        "### Sample chopped and annotated NAIP images"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oJrOGAlF8KBq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "PATH_TO_SAMPLE_IMAGES_DIR = '/content/wind-turbine-detector/images/samples/chopped'\n",
        "SAMPLE_IMAGE_PATHS = glob.glob(os.path.join(PATH_TO_SAMPLE_IMAGES_DIR, \"*.*\"))\n",
        "\n",
        "for image_path in SAMPLE_IMAGE_PATHS:\n",
        "  image = Image.open(image_path)\n",
        "  plt.figure(figsize=(12,8))\n",
        "  plt.axis('off')\n",
        "  plt.imshow(image)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GQGbNpw1oHMf",
        "colab_type": "text"
      },
      "source": [
        "## Training\n",
        "Training will be performed on the 392 labeled images in the train image set and tested against the 80 labeled test images. A pre-trained model from the [TensorFlow Object Detection Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) is used as a starting point. In this notebook, the ssd_inception_v2_coco model is used based on its balance of accuracy and efficiency.  \n",
        "\n",
        "\n",
        "> *Note: If you wish to skip the training step, you can uncomment the second \"PATH_TO_FROZEN_GRAPH=\" line in the **Inference** section to use the provided pre-trained wind turbine detection model.  Training will take ~2.5 hours.*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNvjsjTFoKt4",
        "colab_type": "text"
      },
      "source": [
        "### Install all required libraries\n",
        "Further details on how to install and configure TensorFlow Object Detection API can be found [here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FuQCkr_oVSoR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!apt-get install protobuf-compiler python-pil python-lxml python-tk\n",
        "!pip install Cython\n",
        "!pip install contextlib2\n",
        "!pip install jupyter\n",
        "!pip install matplotlib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-l-vxv7eo4zZ",
        "colab_type": "text"
      },
      "source": [
        "### Clone TensorFlow Object Detection API repo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fr1a5vJeVc3G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone --quiet https://github.com/tensorflow/models.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10Dmf9yJpKgg",
        "colab_type": "text"
      },
      "source": [
        "### COCO API installation\n",
        "This is needed if you are interested in using COCO evaluation metrics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lK-6jvEYg2i3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/cocodataset/cocoapi.git\n",
        "!cd cocoapi/PythonAPI; make; cp -r pycocotools /content/models/research/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BRugsmFnpVmI",
        "colab_type": "text"
      },
      "source": [
        "### Protobuf compilation\n",
        "The Protobuf libraries provided in the TensorFlow Object Detection API repo must be compiled in order to use the framework."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxuEP4WvpqBh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/models/research\n",
        "!protoc object_detection/protos/*.proto --python_out=."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FynXe5k8pyGZ",
        "colab_type": "text"
      },
      "source": [
        "### Add Libraries to PYTHONPATH"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bbbnMD-PVmYe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%set_env PYTHONPATH=$PYTHONPATH:/content/models/research:/content/models/research/slim"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jORkIXPyqH91",
        "colab_type": "text"
      },
      "source": [
        "### Test the installation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CTJR8nBZV1bk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python object_detection/builders/model_builder_test.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DQg2GpyCqSZT",
        "colab_type": "text"
      },
      "source": [
        "### Setup and run TensorBoard\n",
        "TensorBoard provides a visualization of various quantitative metrics such as loss as well as a comparison between prediction vs. groundtruth for a subset of images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ySULkGfPf4U_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/wind-turbine-detector"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Gaa9JqZoQt_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
        "!unzip -o ngrok-stable-linux-amd64.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CJWt_frFoXx2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LOG_DIR = 'training/'\n",
        "get_ipython().system_raw(\n",
        "    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'\n",
        "    .format(LOG_DIR)\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jE1M9kX3ozp9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "get_ipython().system_raw('./ngrok http 6006 &')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xv-waeebsZRH",
        "colab_type": "text"
      },
      "source": [
        "#### Get Tensorboard link\n",
        "Click on the link to launch Tensorboard. It will update once the first checkpoint is saved. The plot of the \"loss_1\" scalar will provide the loss as a function of step, matching what is printed to the screen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yzRpzLFao2FG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! curl -s http:https://localhost:4040/api/tunnels | python3 -c \\\n",
        "    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h99J1SF8sdZv",
        "colab_type": "text"
      },
      "source": [
        "### Train model\n",
        "Train the wind turbine detector model using a modified model_main.py file which includes \"tf.logging.set_verbosity(tf.logging.INFO)\" following the import statements to output the loss every 100 steps. The model configuration is provided in wind-turbine-detector/training/ssd_inception_v2_coco_WTDetector.config. This configuration file uses all the default settings provided in the sample ssd_inception_v2_coco.config file except the following:\n",
        "\n",
        "*   num_classes: 1\n",
        "*   batch_size: 12\n",
        "*   fine_tune_checkpoint: \"pre-trained-model/model.ckpt\"\n",
        "*   train_input_reader: {\n",
        "  tf_record_input_reader {\n",
        "    input_path: \"annotations/train.record\"\n",
        "  }\n",
        "  label_map_path: \"annotations/label_map.pbtxt\"\n",
        "}\n",
        "    * *Note: The 'label_map.pbtxt' file required for training contains 1 class: item {\n",
        "id: 1\n",
        "name: 'wind turbine'\n",
        "}*\n",
        "*   eval_input_reader: {\n",
        "  tf_record_input_reader {\n",
        "    input_path: \"annotations/test.record\"\n",
        "  }\n",
        "  label_map_path: \"annotations/label_map.pbtxt\"\n",
        "  shuffle: false\n",
        "  num_readers: 1\n",
        "}  \n",
        "\n",
        "\n",
        "\n",
        "Additional data (image) augmentation was prescribed in the configuration file. Combining a vertical flip and a 90 degree rotation with the default horizontal flip, the training data can be extended to contain all possible wind turbine orientations. These operations help to generalize the model.\n",
        "*   data_augmentation_options {\n",
        "    random_vertical_flip {\n",
        "    }\n",
        "  }\n",
        "*   data_augmentation_options {\n",
        "    random_rotation90 {\n",
        "    }\n",
        "  }\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rg_XzuZNrTok",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python model_main.py --pipeline_config_path=training/ssd_inception_v2_coco_WTDetector.config --model_dir=training/ --num_train_steps=20000 --alsologtostderr\n",
        "#!python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_inception_v2_coco_WTDetector.config # using legacy training code"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5W2X-Uvf7Sku",
        "colab_type": "text"
      },
      "source": [
        "### Export trained wind turbine detector model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BgrAVZ0WOKhO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python /content/models/research/object_detection/export_inference_graph.py \\\n",
        "    --input_type=image_tensor \\\n",
        "    --pipeline_config_path=training/ssd_inception_v2_coco_WTDetector.config \\\n",
        "    --output_directory=WTDetectorModel \\\n",
        "    --trained_checkpoint_prefix=training/model.ckpt-20000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vBOJ3rs9pZwk",
        "colab_type": "text"
      },
      "source": [
        "## *Inference*\n",
        "\n",
        "Perform inference using the newly trained wind turbine detection model on the validation image set. This set of images was kept separate from the test and train image sets and will now be used to validate the accuracy of the model.   "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dNjT4MhXIR0q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/models/research/object_detection"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t1s3VA-UmIjq",
        "colab_type": "text"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "beJT-03bIq0e",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import os\n",
        "import six.moves.urllib as urllib\n",
        "import sys\n",
        "import tarfile\n",
        "import tensorflow as tf\n",
        "import zipfile\n",
        "\n",
        "from distutils.version import StrictVersion\n",
        "from collections import defaultdict\n",
        "from io import StringIO\n",
        "from matplotlib import pyplot as plt\n",
        "from PIL import Image\n",
        "\n",
        "# This is needed since the notebook is stored in the object_detection folder.\n",
        "sys.path.append(\"..\")\n",
        "from object_detection.utils import ops as utils_ops\n",
        "\n",
        "if StrictVersion(tf.__version__) < StrictVersion('1.9.0'):\n",
        "  raise ImportError('Please upgrade your TensorFlow installation to v1.9.* or later!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLEvyjrDmNKQ",
        "colab_type": "text"
      },
      "source": [
        "#### Env setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AxHY7c2Khgfc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# This is needed to display the images.\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7u3Zdi7mVdb",
        "colab_type": "text"
      },
      "source": [
        "#### Object detection imports\n",
        "Here are the imports from the object detection module."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-B5qni4GhhIe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from utils import label_map_util\n",
        "\n",
        "from utils import visualization_utils as vis_util"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CMg14pzhmcxw",
        "colab_type": "text"
      },
      "source": [
        "### Model preparation "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qdpy94wImdoA",
        "colab_type": "text"
      },
      "source": [
        "#### Variables\n",
        "\n",
        "Any model exported using the \"export_inference_graph.py\" tool can be loaded here simply by changing \"PATH_TO_FROZEN_GRAPH\" to point to a new .pb file.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JElDuCKIhhLK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Path to frozen detection graph. This is the actual model that is used for the object detection.\n",
        "PATH_TO_FROZEN_GRAPH = '/content/wind-turbine-detector/WTDetectorModel/frozen_inference_graph.pb'\n",
        "#PATH_TO_FROZEN_GRAPH = '/content/wind-turbine-detector/trainedWTDetector/frozen_inference_graph.pb' # Uncomment this line to run inference (without training) using provided pre-trained model\n",
        "\n",
        "# List of the strings that is used to add correct label for each box.\n",
        "PATH_TO_LABELS = '/content/wind-turbine-detector/annotations/label_map.pbtxt'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vwO9XmZlmpDY",
        "colab_type": "text"
      },
      "source": [
        "#### Load a (frozen) TensorFlow model into memory"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xHau6ysLi2yv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "detection_graph = tf.Graph()\n",
        "with detection_graph.as_default():\n",
        "  od_graph_def = tf.GraphDef()\n",
        "  with tf.gfile.GFile(PATH_TO_FROZEN_GRAPH, 'rb') as fid:\n",
        "    serialized_graph = fid.read()\n",
        "    od_graph_def.ParseFromString(serialized_graph)\n",
        "    tf.import_graph_def(od_graph_def, name='')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b0VnLMMvmtPp",
        "colab_type": "text"
      },
      "source": [
        "#### Loading label map\n",
        "Label maps map indices to category names, so that when our convolution network predicts `5`, we know that this corresponds to \"airplane\".  Here we use internal utility functions, but anything that returns a dictionary mapping integers to appropriate string labels would be fine"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZofMI3VLi-Vz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)\n",
        "print(category_index)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AMwE_dYXmxPY",
        "colab_type": "text"
      },
      "source": [
        "#### Helper code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVvBib_QjDl9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def load_image_into_numpy_array(image):\n",
        "  (im_width, im_height) = image.size\n",
        "  return np.array(image.getdata()).reshape(\n",
        "      (im_height, im_width, 3)).astype(np.uint8)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mqU0ar1Jm58Q",
        "colab_type": "text"
      },
      "source": [
        "### Detection"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WcYD09DjjR_1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import glob\n",
        "\n",
        "PATH_TO_TEST_IMAGES_DIR = '/content/wind-turbine-detector/images/valid'\n",
        "TEST_IMAGE_PATHS = glob.glob(os.path.join(PATH_TO_TEST_IMAGES_DIR, \"*.*\"))\n",
        "print(TEST_IMAGE_PATHS)\n",
        "\n",
        "# Size, in inches, of the output images.\n",
        "IMAGE_SIZE = (12, 8)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6rUOV4iFj5PR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def run_inference_for_single_image(image, graph):\n",
        "  with graph.as_default():\n",
        "    with tf.Session() as sess:\n",
        "      # Get handles to input and output tensors\n",
        "      ops = tf.get_default_graph().get_operations()\n",
        "      all_tensor_names = {output.name for op in ops for output in op.outputs}\n",
        "      tensor_dict = {}\n",
        "      for key in [\n",
        "          'num_detections', 'detection_boxes', 'detection_scores',\n",
        "          'detection_classes', 'detection_masks'\n",
        "      ]:\n",
        "        tensor_name = key + ':0'\n",
        "        if tensor_name in all_tensor_names:\n",
        "          tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(\n",
        "              tensor_name)\n",
        "      if 'detection_masks' in tensor_dict:\n",
        "        # The following processing is only for single image\n",
        "        detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])\n",
        "        detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])\n",
        "        # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.\n",
        "        real_num_detection = tf.cast(tensor_dict['num_detections'][0], tf.int32)\n",
        "        detection_boxes = tf.slice(detection_boxes, [0, 0], [real_num_detection, -1])\n",
        "        detection_masks = tf.slice(detection_masks, [0, 0, 0], [real_num_detection, -1, -1])\n",
        "        detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n",
        "            detection_masks, detection_boxes, image.shape[0], image.shape[1])\n",
        "        detection_masks_reframed = tf.cast(\n",
        "            tf.greater(detection_masks_reframed, 0.5), tf.uint8)\n",
        "        # Follow the convention by adding back the batch dimension\n",
        "        tensor_dict['detection_masks'] = tf.expand_dims(\n",
        "            detection_masks_reframed, 0)\n",
        "      image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')\n",
        "\n",
        "      # Run inference\n",
        "      output_dict = sess.run(tensor_dict,\n",
        "                             feed_dict={image_tensor: np.expand_dims(image, 0)})\n",
        "\n",
        "      # all outputs are float32 numpy arrays, so convert types as appropriate\n",
        "      output_dict['num_detections'] = int(output_dict['num_detections'][0])\n",
        "      output_dict['detection_classes'] = output_dict[\n",
        "          'detection_classes'][0].astype(np.uint8)\n",
        "      output_dict['detection_boxes'] = output_dict['detection_boxes'][0]\n",
        "      output_dict['detection_scores'] = output_dict['detection_scores'][0]\n",
        "      if 'detection_masks' in output_dict:\n",
        "        output_dict['detection_masks'] = output_dict['detection_masks'][0]\n",
        "  return output_dict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZgbO1Ov8j7rm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for image_path in TEST_IMAGE_PATHS:\n",
        "  image = Image.open(image_path)\n",
        "  # the array based representation of the image will be used later in order to prepare the\n",
        "  # result image with boxes and labels on it.\n",
        "  image_np = load_image_into_numpy_array(image)\n",
        "  # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n",
        "  image_np_expanded = np.expand_dims(image_np, axis=0)\n",
        "  # Actual detection.\n",
        "  output_dict = run_inference_for_single_image(image_np, detection_graph)\n",
        "  # Visualization of the results of a detection.\n",
        "  vis_util.visualize_boxes_and_labels_on_image_array(\n",
        "      image_np,\n",
        "      output_dict['detection_boxes'],\n",
        "      output_dict['detection_classes'],\n",
        "      output_dict['detection_scores'],\n",
        "      category_index,\n",
        "      instance_masks=output_dict.get('detection_masks'),\n",
        "      use_normalized_coordinates=True,\n",
        "      line_thickness=8)\n",
        "  plt.figure(figsize=IMAGE_SIZE)\n",
        "  plt.axis('off')\n",
        "  plt.imshow(image_np)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u8QmWQhTrnJO",
        "colab_type": "text"
      },
      "source": [
        "## Summary\n",
        "\n",
        "The trained model accurately detects 15 out of 17 wind turbines in the validation image set with high probability. This represents an accuracy of ~90% (88.2%). Higher accuracy would likely be acheived by using a larger set of images (train + test) as well as using a more accurate pre-trained model. Alternative models, including those with higher mAP, can be found at the [TensorFlow Object Detection Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md). Details of the trade-offs between speed, accuracy, and memory for various object detection model architectures (Faster RCNN, SSD, R-FCN) can be found in this [paper](https://arxiv.org/pdf/1611.10012.pdf), which can serve as a good starting point in determining which architecture is best for your application."
      ]
    }
  ]
}