diff --git a/WindTurbineDetector_200114.ipynb b/WindTurbineDetector_200114.ipynb new file mode 100644 index 0000000..959c736 --- /dev/null +++ b/WindTurbineDetector_200114.ipynb @@ -0,0 +1,1659 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "WindTurbineDetector_200113.ipynb", + "provenance": [], + "collapsed_sections": [], + "toc_visible": true + }, + "kernelspec": { + "name": "python2", + "display_name": "Python 2" + }, + "accelerator": "GPU" + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "qRSPkQ_ryE9t", + "colab_type": "text" + }, + "source": [ + "# Wind Turbine Object Detection from Aerial Imagery Using TensorFlow Object Detection API and Google Colab" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9vLSXaxZ2aCT", + "colab_type": "text" + }, + "source": [ + "## Introduction\n", + "\n", + "This notebook provides the full pipeline to perform training and inference for a wind turbine object detection model using publicly available aerial images and the [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection). It is designed to run in [Google Colab](https://colab.research.google.com/notebooks/welcome.ipynb), a Jupyter notebook environment running on a virtual machine (VM) that provides free access to a Tesla K80 GPU for up to 12 hours.\n", + "\n", + "\n", + "The aerial image data set used in this notebook is obtained from the [National Agriculture Imagery Program (NAIP) database](https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/) using [USGS EarthExplorer](https://earthexplorer.usgs.gov/). The particular NAIP images used to train, test, and validate this model are from three wind farms located in west-central Iowa containing turbines of varying capacity, style, and manufacturer. A sample NAIP image is presented below in the \"Sample NAIP image\" section. The original NAIP images are 5978 x 7648 so they had to be chipped into smaller individual images to avoid excessive memory use. In addition, the ratio of object size to image size is improved by this operation. An image size of 300 x 300 was chosen since the TensorFlow object detection SSD-based models rescale all input images to this size. \n", + "\n", + "A total of 488 images, all containing at least one full wind turbine, were collected and split into train (\\~80%), test (\\~16%), and validate (\\~4%) sets. [LabelImg](https://github.com/tzutalin/labelImg) was then used to label all the images in the train and test sets. Samples of the chipped and annotated images are shown below in the \"Sample chipped and annotated NAIP images\" section. Annotating the images in LabelImg creates an XML file corresponding to each image. These XML files must be converted to CSV and then TFRecords. Sample code for this can be found [here](https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9) or [here](https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html) (among other places)." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "TahAg7fTspdN", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# clone wind-turbine-detector repo\n", + "!git clone https://github.com/lbborkowski/wind-turbine-detector.git" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NIL8IS9R57U-", + "colab_type": "text" + }, + "source": [ + "### Sample NAIP image" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "wARV6z136BrQ", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from matplotlib import pyplot as plt\n", + "from PIL import Image\n", + "import os\n", + "import glob\n", + "\n", + "%matplotlib inline\n", + "\n", + "image = Image.open('/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg')\n", + "plt.figure(figsize=(12,8))\n", + "plt.axis('off')\n", + "plt.imshow(image)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xjj9v7yf8He4", + "colab_type": "text" + }, + "source": [ + "### Sample chipped and annotated NAIP images" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "oJrOGAlF8KBq", + "colab_type": "code", + "colab": {} + }, + "source": [ + "PATH_TO_SAMPLE_IMAGES_DIR = '/content/wind-turbine-detector/images/samples/chopped'\n", + "SAMPLE_IMAGE_PATHS = glob.glob(os.path.join(PATH_TO_SAMPLE_IMAGES_DIR, \"*.*\"))\n", + "\n", + "for image_path in SAMPLE_IMAGE_PATHS:\n", + " image = Image.open(image_path)\n", + " plt.figure(figsize=(8,8))\n", + " plt.axis('off')\n", + " plt.imshow(image)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GQGbNpw1oHMf", + "colab_type": "text" + }, + "source": [ + "## Training\n", + "Training will be performed on the 392 labeled images in the train image set and tested against the 80 labeled test images. A pre-trained model from the [TensorFlow Object Detection Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md) is used as a starting point. In this notebook, the ssd_inception_v2_coco model is used based on its balance of accuracy and efficiency. \n", + "\n", + "\n", + "> *Note: Training will take ~2.5 hours. If you wish to bypass the training step, you can skip the \"Train model\" and \"Export trained wind turbine detector model\" sections and uncomment the second \"PATH_TO_FROZEN_GRAPH=\" line in the \"Inference\" section to use the provided pre-trained wind turbine detection model.*\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JNvjsjTFoKt4", + "colab_type": "text" + }, + "source": [ + "### Install all required libraries\n", + "Further details on how to install and configure TensorFlow Object Detection API can be found [here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md)." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "FuQCkr_oVSoR", + "colab_type": "code", + "colab": {} + }, + "source": [ + "!apt-get install protobuf-compiler python-pil python-lxml python-tk\n", + "!pip install Cython\n", + "!pip install contextlib2\n", + "!pip install jupyter\n", + "!pip install matplotlib" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-l-vxv7eo4zZ", + "colab_type": "text" + }, + "source": [ + "### Clone TensorFlow Object Detection API repo" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "fr1a5vJeVc3G", + "colab_type": "code", + "colab": {} + }, + "source": [ + "!git clone --quiet https://github.com/tensorflow/models.git" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "10Dmf9yJpKgg", + "colab_type": "text" + }, + "source": [ + "### COCO API installation\n", + "This is needed if you are interested in using COCO evaluation metrics." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "lK-6jvEYg2i3", + "colab_type": "code", + "colab": {} + }, + "source": [ + "!git clone https://github.com/cocodataset/cocoapi.git\n", + "!cd cocoapi/PythonAPI; make; cp -r pycocotools /content/models/research/" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BRugsmFnpVmI", + "colab_type": "text" + }, + "source": [ + "### Protobuf compilation\n", + "The Protobuf libraries provided in the TensorFlow Object Detection API repo must be compiled in order to use the framework." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "zxuEP4WvpqBh", + "colab_type": "code", + "colab": {} + }, + "source": [ + "%cd /content/models/research\n", + "!protoc object_detection/protos/*.proto --python_out=." + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FynXe5k8pyGZ", + "colab_type": "text" + }, + "source": [ + "### Add Libraries to PYTHONPATH" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "bbbnMD-PVmYe", + "colab_type": "code", + "colab": {} + }, + "source": [ + "%set_env PYTHONPATH=$PYTHONPATH:/content/models/research:/content/models/research/slim" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jORkIXPyqH91", + "colab_type": "text" + }, + "source": [ + "### Test the installation" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "CTJR8nBZV1bk", + "colab_type": "code", + "colab": {} + }, + "source": [ + "!python object_detection/builders/model_builder_test.py" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DQg2GpyCqSZT", + "colab_type": "text" + }, + "source": [ + "### Setup and run TensorBoard\n", + "TensorBoard provides a visualization of various quantitative metrics such as loss as well as a comparison between prediction vs. ground truth for a subset of images." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ySULkGfPf4U_", + "colab_type": "code", + "colab": {} + }, + "source": [ + "%cd /content/wind-turbine-detector" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "_Gaa9JqZoQt_", + "colab_type": "code", + "colab": {} + }, + "source": [ + "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n", + "!unzip -o ngrok-stable-linux-amd64.zip" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "CJWt_frFoXx2", + "colab_type": "code", + "colab": {} + }, + "source": [ + "LOG_DIR = 'training/'\n", + "get_ipython().system_raw(\n", + " 'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'\n", + " .format(LOG_DIR)\n", + ")" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "jE1M9kX3ozp9", + "colab_type": "code", + "colab": {} + }, + "source": [ + "get_ipython().system_raw('./ngrok http 6006 &')" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xv-waeebsZRH", + "colab_type": "text" + }, + "source": [ + "#### Get TensorBoard link\n", + "Click on the link to launch TensorBoard. It will update once the first checkpoint is saved. The plot of the \"loss_1\" scalar will provide the loss as a function of step, matching what is printed to the screen." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "yzRpzLFao2FG", + "colab_type": "code", + "colab": {} + }, + "source": [ + "! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n", + " \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "h99J1SF8sdZv", + "colab_type": "text" + }, + "source": [ + "### Train model\n", + "Train the wind turbine detector model using a modified model_main.py file which includes \"tf.logging.set_verbosity(tf.logging.INFO)\" following the import statements to output the loss every 100 steps. The model configuration is provided in wind-turbine-detector/training/ssd_inception_v2_coco_WTDetector.config. This configuration file uses all the default settings provided in the sample ssd_inception_v2_coco.config file except the following:\n", + "\n", + "* num_classes: 1\n", + "* batch_size: 12\n", + "* fine_tune_checkpoint: \"pre-trained-model/model.ckpt\"\n", + "* train_input_reader: {\n", + " tf_record_input_reader {\n", + " input_path: \"annotations/train.record\"\n", + " }\n", + " label_map_path: \"annotations/label_map.pbtxt\"\n", + "}\n", + " * *Note: The 'label_map.pbtxt' file required for training contains 1 class: item {\n", + "id: 1\n", + "name: 'wind turbine'\n", + "}*\n", + "* eval_input_reader: {\n", + " tf_record_input_reader {\n", + " input_path: \"annotations/test.record\"\n", + " }\n", + " label_map_path: \"annotations/label_map.pbtxt\"\n", + " shuffle: false\n", + " num_readers: 1\n", + "} \n", + "\n", + "\n", + "\n", + "Additional data (image) augmentation was prescribed in the configuration file. Combining a vertical flip and a 90 degree rotation with the default horizontal flip, the training data can be extended to contain all possible wind turbine orientations. These operations help to generalize the model.\n", + "* data_augmentation_options {\n", + " random_vertical_flip {\n", + " }\n", + " }\n", + "* data_augmentation_options {\n", + " random_rotation90 {\n", + " }\n", + " }\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Rg_XzuZNrTok", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "outputId": "8bc14cee-2b64-4b71-ae2b-050ce43a1b4a" + }, + "source": [ + "!python model_main.py --pipeline_config_path=training/ssd_inception_v2_coco_WTDetector.config --model_dir=training/ --num_train_steps=20000 --alsologtostderr\n", + "#!python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_inception_v2_coco_WTDetector.config # using legacy training code" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "/content/models/research/object_detection/utils/visualization_utils.py:29: UserWarning: \n", + "This call to matplotlib.use() has no effect because the backend has already\n", + "been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,\n", + "or matplotlib.backends is imported for the first time.\n", + "\n", + "The backend was *originally* set to 'module://ipykernel.pylab.backend_inline' by the following code:\n", + " File \"model_main.py\", line 26, in \n", + " from object_detection import model_lib\n", + " File \"/content/models/research/object_detection/model_lib.py\", line 27, in \n", + " from object_detection import eval_util\n", + " File \"/content/models/research/object_detection/eval_util.py\", line 33, in \n", + " from object_detection.metrics import coco_evaluation\n", + " File \"/content/models/research/object_detection/metrics/coco_evaluation.py\", line 25, in \n", + " from object_detection.metrics import coco_tools\n", + " File \"/content/models/research/object_detection/metrics/coco_tools.py\", line 51, in \n", + " from pycocotools import coco\n", + " File \"/content/models/research/pycocotools/coco.py\", line 49, in \n", + " import matplotlib.pyplot as plt\n", + " File \"/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py\", line 71, in \n", + " from matplotlib.backends import pylab_setup\n", + " File \"/usr/local/lib/python2.7/dist-packages/matplotlib/backends/__init__.py\", line 17, in \n", + " line for line in traceback.format_stack()\n", + "\n", + "\n", + " import matplotlib; matplotlib.use('Agg') # pylint: disable=multiple-statements\n", + "WARNING: Logging before flag parsing goes to stderr.\n", + "W0115 02:13:15.442846 139997462517632 lazy_loader.py:50] \n", + "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n", + "For more information, please see:\n", + " * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n", + " * https://github.com/tensorflow/addons\n", + " * https://github.com/tensorflow/io (for I/O related ops)\n", + "If you depend on functionality not listed there, please file an issue.\n", + "\n", + "W0115 02:13:16.008086 139997462517632 module_wrapper.py:139] From /content/models/research/slim/nets/inception_resnet_v2.py:374: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.\n", + "\n", + "W0115 02:13:16.013695 139997462517632 module_wrapper.py:139] From /content/models/research/slim/nets/mobilenet/mobilenet.py:397: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.\n", + "\n", + "W0115 02:13:16.056530 139997462517632 module_wrapper.py:139] From model_main.py:28: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n", + "\n", + "W0115 02:13:16.056704 139997462517632 module_wrapper.py:139] From model_main.py:28: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n", + "\n", + "W0115 02:13:16.057019 139997462517632 module_wrapper.py:139] From model_main.py:111: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n", + "\n", + "W0115 02:13:16.057621 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/utils/config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n", + "\n", + "W0115 02:13:16.060437 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.\n", + "\n", + "W0115 02:13:16.060570 139997462517632 model_lib.py:629] Forced number of epochs for all eval validations to be 1.\n", + "W0115 02:13:16.060684 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/utils/config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n", + "\n", + "I0115 02:13:16.060754 139997462517632 config_util.py:488] Maybe overwriting use_bfloat16: False\n", + "I0115 02:13:16.060880 139997462517632 config_util.py:488] Maybe overwriting eval_num_epochs: 1\n", + "I0115 02:13:16.061031 139997462517632 config_util.py:488] Maybe overwriting load_pretrained: True\n", + "I0115 02:13:16.061125 139997462517632 config_util.py:498] Ignoring config override key: load_pretrained\n", + "I0115 02:13:16.061217 139997462517632 config_util.py:488] Maybe overwriting train_steps: 20000\n", + "I0115 02:13:16.061292 139997462517632 config_util.py:488] Maybe overwriting sample_1_of_n_eval_examples: 1\n", + "W0115 02:13:16.061412 139997462517632 model_lib.py:645] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.\n", + "I0115 02:13:16.061522 139997462517632 model_lib.py:680] create_estimator_and_inputs: use_tpu False, export_to_tpu False\n", + "I0115 02:13:16.061929 139997462517632 estimator.py:212] Using config: {'_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n", + "graph_options {\n", + " rewrite_options {\n", + " meta_optimizer_iterations: ONE\n", + " }\n", + "}\n", + ", '_keep_checkpoint_max': 5, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': , '_model_dir': 'training/', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_session_creation_timeout_secs': 7200, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_experimental_max_worker_delay_secs': None, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}\n", + "W0115 02:13:16.062093 139997462517632 model_fn.py:630] Estimator's model_fn () includes params argument, but params are not passed to Estimator.\n", + "I0115 02:13:16.062860 139997462517632 estimator_training.py:186] Not using Distribute Coordinator.\n", + "I0115 02:13:16.063117 139997462517632 training.py:612] Running training and evaluation locally (non-distributed).\n", + "I0115 02:13:16.063411 139997462517632 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.\n", + "W0115 02:13:16.067923 139997462517632 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/training/training_util.py:236: initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n", + "W0115 02:13:16.076864 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/data_decoders/tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.\n", + "\n", + "W0115 02:13:16.077061 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/data_decoders/tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.\n", + "\n", + "W0115 02:13:16.088011 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/builders/dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n", + "\n", + "W0115 02:13:16.088768 139997462517632 dataset_builder.py:72] num_readers has been reduced to 1 to match input file shards.\n", + "W0115 02:13:16.093168 139997462517632 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use `tf.data.experimental.parallel_interleave(...)`.\n", + "W0115 02:13:16.093306 139997462517632 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/contrib/data/python/ops/interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.\n", + "W0115 02:13:16.112349 139997462517632 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:155: map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use `tf.data.Dataset.map()\n", + "W0115 02:13:16.116843 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/__init__.py:12: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.\n", + "\n", + "W0115 02:13:17.365386 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.\n", + "\n", + "W0115 02:13:24.953269 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.\n", + "\n", + "W0115 02:13:25.031311 139997462517632 deprecation.py:323] From /content/models/research/object_detection/utils/ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use tf.where in 2.0, which has the same broadcast rule as np.where\n", + "W0115 02:13:27.152256 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n", + "\n", + "W0115 02:13:30.429491 139997462517632 api.py:332] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/operators/control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.\n", + "W0115 02:13:34.653207 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n", + "\n", + "W0115 02:13:34.654140 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.\n", + "\n", + "W0115 02:13:35.077563 139997462517632 deprecation.py:323] From /content/models/research/object_detection/inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use `tf.cast` instead.\n", + "W0115 02:13:36.739481 139997462517632 module_wrapper.py:139] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.\n", + "\n", + "W0115 02:13:37.211522 139997462517632 deprecation.py:323] From /content/models/research/object_detection/builders/dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use `tf.data.Dataset.batch(..., drop_remainder=True)`.\n", + "I0115 02:13:37.222273 139997462517632 estimator.py:1148] Calling model_fn.\n", + "W0115 02:13:37.362340 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/meta_architectures/ssd_meta_arch.py:597: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n", + "\n", + "W0115 02:13:37.365115 139997462517632 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/contrib/layers/python/layers/layers.py:2784: apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Please use `layer.__call__` method instead.\n", + "W0115 02:13:40.206866 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/core/anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.\n", + "\n", + "I0115 02:13:40.214956 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:13:40.240575 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:13:40.265347 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:13:40.290234 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:13:40.315349 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:13:40.340845 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "W0115 02:13:40.368710 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/utils/variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n", + "\n", + "W0115 02:13:40.370079 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/utils/variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.\n", + "\n", + "W0115 02:13:40.375724 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:353: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.\n", + "\n", + "W0115 02:13:41.304969 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/box_coders/faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.\n", + "\n", + "W0115 02:13:42.916977 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/meta_architectures/ssd_meta_arch.py:1163: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n", + "\n", + "W0115 02:13:42.921828 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/core/losses.py:177: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.\n", + "\n", + "W0115 02:13:42.922835 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/core/losses.py:183: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.\n", + "\n", + "W0115 02:13:43.172358 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/meta_architectures/ssd_meta_arch.py:1275: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n", + "\n", + "W0115 02:13:43.175649 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:380: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n", + "\n", + "W0115 02:13:43.175888 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/utils/learning_schedules.py:66: The name tf.train.exponential_decay is deprecated. Please use tf.compat.v1.train.exponential_decay instead.\n", + "\n", + "W0115 02:13:43.182210 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/builders/optimizer_builder.py:47: The name tf.train.RMSPropOptimizer is deprecated. Please use tf.compat.v1.train.RMSPropOptimizer instead.\n", + "\n", + "W0115 02:13:43.182404 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:398: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n", + "\n", + "W0115 02:13:45.101459 139997462517632 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/training/rmsprop.py:119: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Call initializer instance with the dtype argument instead of passing it to the constructor\n", + "W0115 02:13:51.453490 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:515: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n", + "\n", + "W0115 02:13:52.218007 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:519: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.\n", + "\n", + "W0115 02:13:52.218239 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:520: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n", + "\n", + "I0115 02:13:52.218944 139997462517632 estimator.py:1150] Done calling model_fn.\n", + "I0115 02:13:52.219978 139997462517632 basic_session_run_hooks.py:541] Create CheckpointSaverHook.\n", + "I0115 02:13:56.764928 139997462517632 monitored_session.py:240] Graph was finalized.\n", + "2020-01-15 02:13:56.777088: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n", + "2020-01-15 02:13:56.777308: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55aa81e0e1c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n", + "2020-01-15 02:13:56.777342: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version\n", + "2020-01-15 02:13:56.782524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n", + "2020-01-15 02:13:56.943884: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:13:56.944650: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55aa81e0e380 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", + "2020-01-15 02:13:56.944681: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0\n", + "2020-01-15 02:13:56.946432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:13:56.947029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n", + "name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n", + "pciBusID: 0000:00:04.0\n", + "2020-01-15 02:13:56.961823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n", + "2020-01-15 02:13:57.220733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n", + "2020-01-15 02:13:57.329967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n", + "2020-01-15 02:13:57.372597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n", + "2020-01-15 02:13:57.629752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n", + "2020-01-15 02:13:57.667214: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n", + "2020-01-15 02:13:58.153432: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n", + "2020-01-15 02:13:58.153664: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:13:58.154370: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:13:58.155004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n", + "2020-01-15 02:13:58.158676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n", + "2020-01-15 02:13:58.160153: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\n", + "2020-01-15 02:13:58.160186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 \n", + "2020-01-15 02:13:58.160198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N \n", + "2020-01-15 02:13:58.167627: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:13:58.168220: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:13:58.168749: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n", + "2020-01-15 02:13:58.168792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n", + "I0115 02:14:09.626234 139997462517632 session_manager.py:500] Running local_init_op.\n", + "I0115 02:14:10.017987 139997462517632 session_manager.py:502] Done running local_init_op.\n", + "I0115 02:14:21.924941 139997462517632 basic_session_run_hooks.py:606] Saving checkpoints for 0 into training/model.ckpt.\n", + "2020-01-15 02:14:31.986057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n", + "2020-01-15 02:14:36.427272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n", + "I0115 02:14:39.064117 139997462517632 basic_session_run_hooks.py:262] loss = 14.153067, step = 0\n", + "I0115 02:15:02.919287 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 4.19179\n", + "I0115 02:15:02.920181 139997462517632 basic_session_run_hooks.py:260] loss = 6.3255014, step = 100 (23.856 sec)\n", + "I0115 02:15:22.063154 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22363\n", + "I0115 02:15:22.064197 139997462517632 basic_session_run_hooks.py:260] loss = 5.1623216, step = 200 (19.144 sec)\n", + "I0115 02:15:41.115128 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.24876\n", + "I0115 02:15:41.115901 139997462517632 basic_session_run_hooks.py:260] loss = 5.6312075, step = 300 (19.052 sec)\n", + "I0115 02:16:00.370217 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19345\n", + "I0115 02:16:00.371459 139997462517632 basic_session_run_hooks.py:260] loss = 5.1535807, step = 400 (19.256 sec)\n", + "I0115 02:16:19.557734 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21171\n", + "I0115 02:16:19.558949 139997462517632 basic_session_run_hooks.py:260] loss = 4.281559, step = 500 (19.187 sec)\n", + "I0115 02:16:38.803657 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19592\n", + "I0115 02:16:38.804596 139997462517632 basic_session_run_hooks.py:260] loss = 4.6280417, step = 600 (19.246 sec)\n", + "I0115 02:16:58.003910 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20825\n", + "I0115 02:16:58.004899 139997462517632 basic_session_run_hooks.py:260] loss = 7.168648, step = 700 (19.200 sec)\n", + "I0115 02:17:17.198386 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20983\n", + "I0115 02:17:17.199376 139997462517632 basic_session_run_hooks.py:260] loss = 4.190527, step = 800 (19.194 sec)\n", + "I0115 02:17:36.241945 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.25113\n", + "I0115 02:17:36.242949 139997462517632 basic_session_run_hooks.py:260] loss = 3.672969, step = 900 (19.044 sec)\n", + "I0115 02:17:55.303195 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.24624\n", + "I0115 02:17:55.304290 139997462517632 basic_session_run_hooks.py:260] loss = 4.3698015, step = 1000 (19.061 sec)\n", + "I0115 02:18:14.386034 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.24031\n", + "I0115 02:18:14.386920 139997462517632 basic_session_run_hooks.py:260] loss = 3.6316345, step = 1100 (19.083 sec)\n", + "I0115 02:18:33.504615 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23052\n", + "I0115 02:18:33.505672 139997462517632 basic_session_run_hooks.py:260] loss = 4.7542977, step = 1200 (19.119 sec)\n", + "I0115 02:18:52.699537 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20971\n", + "I0115 02:18:52.700592 139997462517632 basic_session_run_hooks.py:260] loss = 7.2705708, step = 1300 (19.195 sec)\n", + "I0115 02:19:11.852164 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.2212\n", + "I0115 02:19:11.853250 139997462517632 basic_session_run_hooks.py:260] loss = 2.836309, step = 1400 (19.153 sec)\n", + "I0115 02:19:30.962446 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23279\n", + "I0115 02:19:30.963635 139997462517632 basic_session_run_hooks.py:260] loss = 4.7855754, step = 1500 (19.110 sec)\n", + "I0115 02:19:50.046181 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.24006\n", + "I0115 02:19:50.047158 139997462517632 basic_session_run_hooks.py:260] loss = 4.2163033, step = 1600 (19.084 sec)\n", + "I0115 02:20:09.221126 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21514\n", + "I0115 02:20:09.222281 139997462517632 basic_session_run_hooks.py:260] loss = 3.0023806, step = 1700 (19.175 sec)\n", + "I0115 02:20:28.293056 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.24331\n", + "I0115 02:20:28.294019 139997462517632 basic_session_run_hooks.py:260] loss = 4.013126, step = 1800 (19.072 sec)\n", + "I0115 02:20:47.297396 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.26196\n", + "I0115 02:20:47.298408 139997462517632 basic_session_run_hooks.py:260] loss = 3.5170317, step = 1900 (19.004 sec)\n", + "I0115 02:21:06.293020 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.26437\n", + "I0115 02:21:06.293982 139997462517632 basic_session_run_hooks.py:260] loss = 3.2588515, step = 2000 (18.996 sec)\n", + "I0115 02:21:25.547734 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19354\n", + "I0115 02:21:25.548903 139997462517632 basic_session_run_hooks.py:260] loss = 3.8542557, step = 2100 (19.255 sec)\n", + "I0115 02:21:44.746892 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20856\n", + "I0115 02:21:44.747843 139997462517632 basic_session_run_hooks.py:260] loss = 3.407043, step = 2200 (19.199 sec)\n", + "I0115 02:22:03.837939 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23805\n", + "I0115 02:22:03.838804 139997462517632 basic_session_run_hooks.py:260] loss = 3.2069585, step = 2300 (19.091 sec)\n", + "I0115 02:22:22.949852 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23234\n", + "I0115 02:22:22.950721 139997462517632 basic_session_run_hooks.py:260] loss = 4.142082, step = 2400 (19.112 sec)\n", + "I0115 02:22:42.063075 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23198\n", + "I0115 02:22:42.064399 139997462517632 basic_session_run_hooks.py:260] loss = 3.4361515, step = 2500 (19.114 sec)\n", + "I0115 02:23:01.135632 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.24315\n", + "I0115 02:23:01.136595 139997462517632 basic_session_run_hooks.py:260] loss = 2.5690036, step = 2600 (19.072 sec)\n", + "I0115 02:23:20.262260 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22831\n", + "I0115 02:23:20.263430 139997462517632 basic_session_run_hooks.py:260] loss = 2.3147092, step = 2700 (19.127 sec)\n", + "I0115 02:23:39.444763 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21307\n", + "I0115 02:23:39.445666 139997462517632 basic_session_run_hooks.py:260] loss = 4.1876526, step = 2800 (19.182 sec)\n", + "I0115 02:23:58.542001 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23637\n", + "I0115 02:23:58.543009 139997462517632 basic_session_run_hooks.py:260] loss = 4.649087, step = 2900 (19.097 sec)\n", + "I0115 02:24:17.672642 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22722\n", + "I0115 02:24:17.673671 139997462517632 basic_session_run_hooks.py:260] loss = 2.8713658, step = 3000 (19.131 sec)\n", + "I0115 02:24:25.323766 139997462517632 basic_session_run_hooks.py:606] Saving checkpoints for 3041 into training/model.ckpt.\n", + "I0115 02:24:28.575949 139997462517632 estimator.py:1148] Calling model_fn.\n", + "I0115 02:24:30.820779 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:24:30.849965 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:24:30.875668 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:24:30.901686 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:24:30.927826 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:24:30.954773 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "W0115 02:24:31.541389 139997462517632 deprecation.py:323] From /content/models/research/object_detection/eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "Use `tf.cast` instead.\n", + "W0115 02:24:31.706724 139997462517632 deprecation.py:323] From /content/models/research/object_detection/utils/visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "tf.py_func is deprecated in TF V2. Instead, there are two\n", + " options available in V2.\n", + " - tf.py_function takes a python function which manipulates tf eager\n", + " tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n", + " an ndarray (just call tensor.numpy()) but having access to eager tensors\n", + " means `tf.py_function`s can use accelerators such as GPUs as well as\n", + " being differentiable using a gradient tape.\n", + " - tf.numpy_function maintains the semantics of the deprecated tf.py_func\n", + " (it is not differentiable, and manipulates numpy arrays). It drops the\n", + " stateful argument making all functions stateful.\n", + " \n", + "W0115 02:24:31.834111 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/utils/visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.\n", + "\n", + "W0115 02:24:31.903451 139997462517632 module_wrapper.py:139] From /content/models/research/object_detection/model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.\n", + "\n", + "I0115 02:24:32.219208 139997462517632 estimator.py:1150] Done calling model_fn.\n", + "I0115 02:24:32.233129 139997462517632 evaluation.py:255] Starting evaluation at 2020-01-15T02:24:32Z\n", + "I0115 02:24:32.671688 139997462517632 monitored_session.py:240] Graph was finalized.\n", + "2020-01-15 02:24:32.672678: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:24:32.673161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n", + "name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n", + "pciBusID: 0000:00:04.0\n", + "2020-01-15 02:24:32.673255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n", + "2020-01-15 02:24:32.673285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n", + "2020-01-15 02:24:32.673312: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n", + "2020-01-15 02:24:32.673337: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n", + "2020-01-15 02:24:32.673360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n", + "2020-01-15 02:24:32.673387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n", + "2020-01-15 02:24:32.673409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n", + "2020-01-15 02:24:32.673506: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:24:32.673981: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:24:32.674419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n", + "2020-01-15 02:24:32.674465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\n", + "2020-01-15 02:24:32.674497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 \n", + "2020-01-15 02:24:32.674507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N \n", + "2020-01-15 02:24:32.674598: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:24:32.675058: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:24:32.675531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n", + "I0115 02:24:32.676512 139997462517632 saver.py:1284] Restoring parameters from training/model.ckpt-3041\n", + "I0115 02:24:33.769011 139997462517632 session_manager.py:500] Running local_init_op.\n", + "I0115 02:24:33.916573 139997462517632 session_manager.py:502] Done running local_init_op.\n", + "I0115 02:24:38.595581 139995442464512 coco_evaluation.py:205] Performing evaluation on 80 images.\n", + "creating index...\n", + "index created!\n", + "I0115 02:24:38.596271 139995442464512 coco_tools.py:115] Loading and preparing annotation results...\n", + "I0115 02:24:38.602180 139995442464512 coco_tools.py:137] DONE (t=0.01s)\n", + "creating index...\n", + "index created!\n", + "Running per image evaluation...\n", + "Evaluate annotation type *bbox*\n", + "DONE (t=0.60s).\n", + "Accumulating evaluation results...\n", + "DONE (t=0.04s).\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.471\n", + " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.981\n", + " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.362\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.437\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.477\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.507\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.560\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.574\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.450\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.581\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n", + "I0115 02:24:39.347790 139997462517632 evaluation.py:275] Finished evaluation at 2020-01-15-02:24:39\n", + "I0115 02:24:39.348030 139997462517632 estimator.py:2049] Saving dict for global step 3041: DetectionBoxes_Precision/mAP = 0.47144905, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.47668728, DetectionBoxes_Precision/mAP (small) = 0.43663368, DetectionBoxes_Precision/mAP@.50IOU = 0.9807086, DetectionBoxes_Precision/mAP@.75IOU = 0.3623588, DetectionBoxes_Recall/AR@1 = 0.50731707, DetectionBoxes_Recall/AR@10 = 0.5597561, DetectionBoxes_Recall/AR@100 = 0.57439023, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.58076924, DetectionBoxes_Recall/AR@100 (small) = 0.45, Loss/classification_loss = 3.0850036, Loss/localization_loss = 0.8636196, Loss/regularization_loss = 0.5465557, Loss/total_loss = 4.495179, global_step = 3041, learning_rate = 0.004, loss = 4.495179\n", + "I0115 02:24:40.614639 139997462517632 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3041: training/model.ckpt-3041\n", + "I0115 02:24:52.170878 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 2.89869\n", + "I0115 02:24:52.171852 139997462517632 basic_session_run_hooks.py:260] loss = 3.9301276, step = 3100 (34.498 sec)\n", + "I0115 02:25:11.389133 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20339\n", + "I0115 02:25:11.390439 139997462517632 basic_session_run_hooks.py:260] loss = 3.152392, step = 3200 (19.219 sec)\n", + "I0115 02:25:30.675358 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.18505\n", + "I0115 02:25:30.676506 139997462517632 basic_session_run_hooks.py:260] loss = 3.8908706, step = 3300 (19.286 sec)\n", + "I0115 02:25:49.772547 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23638\n", + "I0115 02:25:49.773685 139997462517632 basic_session_run_hooks.py:260] loss = 3.8775303, step = 3400 (19.097 sec)\n", + "I0115 02:26:08.892019 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23026\n", + "I0115 02:26:08.893081 139997462517632 basic_session_run_hooks.py:260] loss = 3.2722871, step = 3500 (19.119 sec)\n", + "I0115 02:26:28.102067 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20561\n", + "I0115 02:26:28.103260 139997462517632 basic_session_run_hooks.py:260] loss = 2.3988602, step = 3600 (19.210 sec)\n", + "I0115 02:26:47.450073 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.16849\n", + "I0115 02:26:47.451431 139997462517632 basic_session_run_hooks.py:260] loss = 2.6166406, step = 3700 (19.348 sec)\n", + "I0115 02:27:07.005947 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.11355\n", + "I0115 02:27:07.007180 139997462517632 basic_session_run_hooks.py:260] loss = 4.037521, step = 3800 (19.556 sec)\n", + "I0115 02:27:26.524391 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.12336\n", + "I0115 02:27:26.525496 139997462517632 basic_session_run_hooks.py:260] loss = 2.9216526, step = 3900 (19.518 sec)\n", + "I0115 02:27:45.941700 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.15004\n", + "I0115 02:27:45.942835 139997462517632 basic_session_run_hooks.py:260] loss = 2.860343, step = 4000 (19.417 sec)\n", + "I0115 02:28:05.309803 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.16313\n", + "I0115 02:28:05.310851 139997462517632 basic_session_run_hooks.py:260] loss = 3.4992654, step = 4100 (19.368 sec)\n", + "I0115 02:28:24.510658 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.2081\n", + "I0115 02:28:24.511620 139997462517632 basic_session_run_hooks.py:260] loss = 4.014844, step = 4200 (19.201 sec)\n", + "I0115 02:28:43.712706 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20778\n", + "I0115 02:28:43.713677 139997462517632 basic_session_run_hooks.py:260] loss = 3.8467786, step = 4300 (19.202 sec)\n", + "I0115 02:29:02.976680 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19103\n", + "I0115 02:29:02.977585 139997462517632 basic_session_run_hooks.py:260] loss = 2.6345162, step = 4400 (19.264 sec)\n", + "I0115 02:29:22.326344 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.16805\n", + "I0115 02:29:22.327419 139997462517632 basic_session_run_hooks.py:260] loss = 3.907684, step = 4500 (19.350 sec)\n", + "I0115 02:29:41.671763 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.16919\n", + "I0115 02:29:41.672712 139997462517632 basic_session_run_hooks.py:260] loss = 2.6832395, step = 4600 (19.345 sec)\n", + "I0115 02:30:00.848829 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21456\n", + "I0115 02:30:00.849860 139997462517632 basic_session_run_hooks.py:260] loss = 2.926046, step = 4700 (19.177 sec)\n", + "I0115 02:30:20.247844 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.15491\n", + "I0115 02:30:20.248694 139997462517632 basic_session_run_hooks.py:260] loss = 3.6159725, step = 4800 (19.399 sec)\n", + "I0115 02:30:39.433790 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21215\n", + "I0115 02:30:39.435058 139997462517632 basic_session_run_hooks.py:260] loss = 3.8733249, step = 4900 (19.186 sec)\n", + "I0115 02:30:58.667521 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.1992\n", + "I0115 02:30:58.668557 139997462517632 basic_session_run_hooks.py:260] loss = 3.4985542, step = 5000 (19.233 sec)\n", + "I0115 02:31:17.918767 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19446\n", + "I0115 02:31:17.919683 139997462517632 basic_session_run_hooks.py:260] loss = 2.6325831, step = 5100 (19.251 sec)\n", + "I0115 02:31:37.145104 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.2012\n", + "I0115 02:31:37.146053 139997462517632 basic_session_run_hooks.py:260] loss = 2.3890295, step = 5200 (19.226 sec)\n", + "I0115 02:31:56.462240 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.1768\n", + "I0115 02:31:56.463339 139997462517632 basic_session_run_hooks.py:260] loss = 2.312985, step = 5300 (19.317 sec)\n", + "I0115 02:32:15.606892 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22335\n", + "I0115 02:32:15.608231 139997462517632 basic_session_run_hooks.py:260] loss = 2.7440426, step = 5400 (19.145 sec)\n", + "I0115 02:32:34.927025 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.17595\n", + "I0115 02:32:34.928138 139997462517632 basic_session_run_hooks.py:260] loss = 2.4785872, step = 5500 (19.320 sec)\n", + "I0115 02:32:54.240173 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.17782\n", + "I0115 02:32:54.241331 139997462517632 basic_session_run_hooks.py:260] loss = 3.1438718, step = 5600 (19.313 sec)\n", + "I0115 02:33:13.669991 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.14672\n", + "I0115 02:33:13.671024 139997462517632 basic_session_run_hooks.py:260] loss = 3.9234018, step = 5700 (19.430 sec)\n", + "I0115 02:33:33.004681 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.17205\n", + "I0115 02:33:33.005796 139997462517632 basic_session_run_hooks.py:260] loss = 2.9255424, step = 5800 (19.335 sec)\n", + "I0115 02:33:52.271641 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19023\n", + "I0115 02:33:52.272815 139997462517632 basic_session_run_hooks.py:260] loss = 2.8107712, step = 5900 (19.267 sec)\n", + "I0115 02:34:11.523780 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19423\n", + "I0115 02:34:11.524703 139997462517632 basic_session_run_hooks.py:260] loss = 2.6977167, step = 6000 (19.252 sec)\n", + "I0115 02:34:25.394438 139997462517632 basic_session_run_hooks.py:606] Saving checkpoints for 6073 into training/model.ckpt.\n", + "I0115 02:34:28.543903 139997462517632 estimator.py:1148] Calling model_fn.\n", + "I0115 02:34:30.807387 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:34:30.834209 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:34:30.861582 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:34:30.888289 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:34:30.914697 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:34:30.942167 139997462517632 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0\n", + "I0115 02:34:32.174312 139997462517632 estimator.py:1150] Done calling model_fn.\n", + "I0115 02:34:32.187911 139997462517632 evaluation.py:255] Starting evaluation at 2020-01-15T02:34:32Z\n", + "I0115 02:34:32.624373 139997462517632 monitored_session.py:240] Graph was finalized.\n", + "2020-01-15 02:34:32.627845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:34:32.628354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n", + "name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n", + "pciBusID: 0000:00:04.0\n", + "2020-01-15 02:34:32.630703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n", + "2020-01-15 02:34:32.630786: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n", + "2020-01-15 02:34:32.630833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n", + "2020-01-15 02:34:32.630865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n", + "2020-01-15 02:34:32.630889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n", + "2020-01-15 02:34:32.630909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n", + "2020-01-15 02:34:32.630930: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n", + "2020-01-15 02:34:32.631034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:34:32.631578: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:34:32.632019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n", + "2020-01-15 02:34:32.632102: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\n", + "2020-01-15 02:34:32.632117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 \n", + "2020-01-15 02:34:32.632137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N \n", + "2020-01-15 02:34:32.632258: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:34:32.632990: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2020-01-15 02:34:32.633460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n", + "I0115 02:34:32.634810 139997462517632 saver.py:1284] Restoring parameters from training/model.ckpt-6073\n", + "I0115 02:34:33.747961 139997462517632 session_manager.py:500] Running local_init_op.\n", + "I0115 02:34:33.906708 139997462517632 session_manager.py:502] Done running local_init_op.\n", + "I0115 02:34:38.342346 139995442464512 coco_evaluation.py:205] Performing evaluation on 80 images.\n", + "creating index...\n", + "index created!\n", + "I0115 02:34:38.342886 139995442464512 coco_tools.py:115] Loading and preparing annotation results...\n", + "I0115 02:34:38.350368 139995442464512 coco_tools.py:137] DONE (t=0.01s)\n", + "creating index...\n", + "index created!\n", + "Running per image evaluation...\n", + "Evaluate annotation type *bbox*\n", + "DONE (t=0.62s).\n", + "Accumulating evaluation results...\n", + "DONE (t=0.04s).\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.562\n", + " Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.966\n", + " Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.598\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.410\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.571\n", + " Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.600\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.643\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.646\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.450\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.656\n", + " Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000\n", + "I0115 02:34:39.114861 139997462517632 evaluation.py:275] Finished evaluation at 2020-01-15-02:34:39\n", + "I0115 02:34:39.115099 139997462517632 estimator.py:2049] Saving dict for global step 6073: DetectionBoxes_Precision/mAP = 0.56238735, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.57126546, DetectionBoxes_Precision/mAP (small) = 0.41023102, DetectionBoxes_Precision/mAP@.50IOU = 0.9664158, DetectionBoxes_Precision/mAP@.75IOU = 0.59802514, DetectionBoxes_Recall/AR@1 = 0.6, DetectionBoxes_Recall/AR@10 = 0.6426829, DetectionBoxes_Recall/AR@100 = 0.64634144, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.6564103, DetectionBoxes_Recall/AR@100 (small) = 0.45, Loss/classification_loss = 3.2361705, Loss/localization_loss = 0.641889, Loss/regularization_loss = 0.5507954, Loss/total_loss = 4.428854, global_step = 6073, learning_rate = 0.004, loss = 4.428854\n", + "I0115 02:34:39.118614 139997462517632 estimator.py:2109] Saving 'checkpoint_path' summary for global step 6073: training/model.ckpt-6073\n", + "I0115 02:34:44.496094 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 3.03285\n", + "I0115 02:34:44.497004 139997462517632 basic_session_run_hooks.py:260] loss = 3.02704, step = 6100 (32.972 sec)\n", + "I0115 02:35:03.650650 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22069\n", + "I0115 02:35:03.651711 139997462517632 basic_session_run_hooks.py:260] loss = 2.3249722, step = 6200 (19.155 sec)\n", + "I0115 02:35:22.898808 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19531\n", + "I0115 02:35:22.899931 139997462517632 basic_session_run_hooks.py:260] loss = 3.8510346, step = 6300 (19.248 sec)\n", + "I0115 02:35:42.108578 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20569\n", + "I0115 02:35:42.109572 139997462517632 basic_session_run_hooks.py:260] loss = 2.8015418, step = 6400 (19.210 sec)\n", + "I0115 02:36:01.274574 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21757\n", + "I0115 02:36:01.275849 139997462517632 basic_session_run_hooks.py:260] loss = 3.6884418, step = 6500 (19.166 sec)\n", + "I0115 02:36:20.601208 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.17421\n", + "I0115 02:36:20.602119 139997462517632 basic_session_run_hooks.py:260] loss = 3.0049162, step = 6600 (19.326 sec)\n", + "I0115 02:36:39.801429 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20827\n", + "I0115 02:36:39.802510 139997462517632 basic_session_run_hooks.py:260] loss = 2.753242, step = 6700 (19.200 sec)\n", + "I0115 02:36:58.902359 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23535\n", + "I0115 02:36:58.903439 139997462517632 basic_session_run_hooks.py:260] loss = 2.394912, step = 6800 (19.101 sec)\n", + "I0115 02:37:18.180949 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.1871\n", + "I0115 02:37:18.182173 139997462517632 basic_session_run_hooks.py:260] loss = 2.1665902, step = 6900 (19.279 sec)\n", + "I0115 02:37:37.397197 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20394\n", + "I0115 02:37:37.398025 139997462517632 basic_session_run_hooks.py:260] loss = 2.947222, step = 7000 (19.216 sec)\n", + "I0115 02:37:56.641246 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19641\n", + "I0115 02:37:56.642148 139997462517632 basic_session_run_hooks.py:260] loss = 2.0622702, step = 7100 (19.244 sec)\n", + "I0115 02:38:15.902695 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19172\n", + "I0115 02:38:15.903695 139997462517632 basic_session_run_hooks.py:260] loss = 2.2884507, step = 7200 (19.262 sec)\n", + "I0115 02:38:35.140410 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19813\n", + "I0115 02:38:35.141490 139997462517632 basic_session_run_hooks.py:260] loss = 3.579755, step = 7300 (19.238 sec)\n", + "I0115 02:38:54.489914 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.16808\n", + "I0115 02:38:54.490813 139997462517632 basic_session_run_hooks.py:260] loss = 2.2656393, step = 7400 (19.349 sec)\n", + "I0115 02:39:13.871931 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.15942\n", + "I0115 02:39:13.872844 139997462517632 basic_session_run_hooks.py:260] loss = 3.1827636, step = 7500 (19.382 sec)\n", + "I0115 02:39:33.104095 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19962\n", + "I0115 02:39:33.105343 139997462517632 basic_session_run_hooks.py:260] loss = 2.5550041, step = 7600 (19.232 sec)\n", + "I0115 02:39:52.432225 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.17382\n", + "I0115 02:39:52.433516 139997462517632 basic_session_run_hooks.py:260] loss = 3.0342543, step = 7700 (19.328 sec)\n", + "I0115 02:40:11.659697 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20088\n", + "I0115 02:40:11.660773 139997462517632 basic_session_run_hooks.py:260] loss = 2.823333, step = 7800 (19.227 sec)\n", + "I0115 02:40:30.887491 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20081\n", + "I0115 02:40:30.888681 139997462517632 basic_session_run_hooks.py:260] loss = 2.1274822, step = 7900 (19.228 sec)\n", + "I0115 02:40:50.109229 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.20243\n", + "I0115 02:40:50.110122 139997462517632 basic_session_run_hooks.py:260] loss = 2.1046238, step = 8000 (19.221 sec)\n", + "I0115 02:41:09.249666 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22455\n", + "I0115 02:41:09.250870 139997462517632 basic_session_run_hooks.py:260] loss = 2.7200673, step = 8100 (19.141 sec)\n", + "I0115 02:41:28.432780 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21292\n", + "I0115 02:41:28.433779 139997462517632 basic_session_run_hooks.py:260] loss = 2.260762, step = 8200 (19.183 sec)\n", + "I0115 02:41:47.459249 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.25585\n", + "I0115 02:41:47.460499 139997462517632 basic_session_run_hooks.py:260] loss = 2.3165793, step = 8300 (19.027 sec)\n", + "I0115 02:42:06.576093 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23098\n", + "I0115 02:42:06.577302 139997462517632 basic_session_run_hooks.py:260] loss = 2.2381597, step = 8400 (19.117 sec)\n", + "I0115 02:42:25.879506 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.18044\n", + "I0115 02:42:25.880465 139997462517632 basic_session_run_hooks.py:260] loss = 2.8980827, step = 8500 (19.303 sec)\n", + "I0115 02:42:45.111839 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19958\n", + "I0115 02:42:45.113172 139997462517632 basic_session_run_hooks.py:260] loss = 2.9196088, step = 8600 (19.233 sec)\n", + "I0115 02:43:04.582562 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.13591\n", + "I0115 02:43:04.583600 139997462517632 basic_session_run_hooks.py:260] loss = 3.5013695, step = 8700 (19.470 sec)\n", + "I0115 02:43:23.906929 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.17481\n", + "I0115 02:43:23.908174 139997462517632 basic_session_run_hooks.py:260] loss = 2.46058, step = 8800 (19.325 sec)\n", + "I0115 02:43:43.028155 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22979\n", + "I0115 02:43:43.029325 139997462517632 basic_session_run_hooks.py:260] loss = 2.9245834, step = 8900 (19.121 sec)\n", + "I0115 02:44:02.273416 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19609\n", + "I0115 02:44:02.274558 139997462517632 basic_session_run_hooks.py:260] loss = 3.2697775, step = 9000 (19.245 sec)\n", + "I0115 02:44:21.564925 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.18362\n", + "I0115 02:44:21.566097 139997462517632 basic_session_run_hooks.py:260] loss = 3.383832, step = 9100 (19.292 sec)\n", + "I0115 02:44:25.406497 139997462517632 basic_session_run_hooks.py:606] Saving checkpoints for 9121 into training/model.ckpt.\n", + "I0115 02:44:27.944293 139997462517632 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).\n", + "I0115 02:44:43.317995 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 4.59705\n", + "I0115 02:44:43.319122 139997462517632 basic_session_run_hooks.py:260] loss = 2.1714542, step = 9200 (21.753 sec)\n", + "I0115 02:45:02.423547 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.23409\n", + "I0115 02:45:02.424585 139997462517632 basic_session_run_hooks.py:260] loss = 1.949295, step = 9300 (19.105 sec)\n", + "I0115 02:45:21.606920 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.21284\n", + "I0115 02:45:21.607856 139997462517632 basic_session_run_hooks.py:260] loss = 1.8717009, step = 9400 (19.183 sec)\n", + "I0115 02:45:40.858642 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.19435\n", + "I0115 02:45:40.859611 139997462517632 basic_session_run_hooks.py:260] loss = 2.8430595, step = 9500 (19.252 sec)\n", + "I0115 02:46:00.201916 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.16975\n", + "I0115 02:46:00.203054 139997462517632 basic_session_run_hooks.py:260] loss = 2.4040422, step = 9600 (19.343 sec)\n", + "I0115 02:46:19.353028 139997462517632 basic_session_run_hooks.py:692] global_step/sec: 5.22163\n", + "I0115 02:46:19.354310 139997462517632 basic_session_run_hooks.py:260] loss = 2.7041926, step = 9700 (19.151 sec)\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5W2X-Uvf7Sku", + "colab_type": "text" + }, + "source": [ + "### Export trained wind turbine detector model" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "BgrAVZ0WOKhO", + "colab_type": "code", + "colab": {} + }, + "source": [ + "!python /content/models/research/object_detection/export_inference_graph.py \\\n", + " --input_type=image_tensor \\\n", + " --pipeline_config_path=training/ssd_inception_v2_coco_WTDetector.config \\\n", + " --output_directory=WTDetectorModel \\\n", + " --trained_checkpoint_prefix=training/model.ckpt-20000" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vBOJ3rs9pZwk", + "colab_type": "text" + }, + "source": [ + "## *Inference*\n", + "\n", + "Perform inference using the newly trained wind turbine detection model on the validation image set. This set of images was kept separate from the test and train image sets and will now be used to validate the accuracy of the model. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "dNjT4MhXIR0q", + "colab_type": "code", + "colab": {} + }, + "source": [ + "%cd /content/models/research/object_detection" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t1s3VA-UmIjq", + "colab_type": "text" + }, + "source": [ + "### Imports" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "beJT-03bIq0e", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import numpy as np\n", + "import os\n", + "import six.moves.urllib as urllib\n", + "import sys\n", + "import tarfile\n", + "import tensorflow as tf\n", + "import zipfile\n", + "\n", + "from distutils.version import StrictVersion\n", + "from collections import defaultdict\n", + "from io import StringIO\n", + "from matplotlib import pyplot as plt\n", + "from PIL import Image\n", + "\n", + "# This is needed since the notebook is stored in the object_detection folder.\n", + "sys.path.append(\"..\")\n", + "from object_detection.utils import ops as utils_ops\n", + "\n", + "#if StrictVersion(tf.__version__) < StrictVersion('1.9.0'):\n", + "# raise ImportError('Please upgrade your TensorFlow installation to v1.9.* or later!')" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iLEvyjrDmNKQ", + "colab_type": "text" + }, + "source": [ + "#### Env setup" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "AxHY7c2Khgfc", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# This is needed to display the images.\n", + "%matplotlib inline" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "z7u3Zdi7mVdb", + "colab_type": "text" + }, + "source": [ + "#### Object detection imports\n", + "Here are the imports from the object detection module." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "-B5qni4GhhIe", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from utils import label_map_util\n", + "\n", + "from utils import visualization_utils as vis_util" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CMg14pzhmcxw", + "colab_type": "text" + }, + "source": [ + "### Model preparation " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Qdpy94wImdoA", + "colab_type": "text" + }, + "source": [ + "#### Variables\n", + "\n", + "Any model exported using the \"export_inference_graph.py\" tool can be loaded here simply by changing \"PATH_TO_FROZEN_GRAPH\" to point to a new .pb file. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "JElDuCKIhhLK", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Path to frozen detection graph. This is the actual model that is used for the object detection.\n", + "PATH_TO_FROZEN_GRAPH = '/content/wind-turbine-detector/WTDetectorModel/frozen_inference_graph.pb'\n", + "#PATH_TO_FROZEN_GRAPH = '/content/wind-turbine-detector/trainedWTDetector/frozen_inference_graph.pb' # Uncomment this line to run inference (without training) using provided pre-trained model\n", + "\n", + "# List of the strings that is used to add correct label for each box.\n", + "PATH_TO_LABELS = '/content/wind-turbine-detector/annotations/label_map.pbtxt'" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vwO9XmZlmpDY", + "colab_type": "text" + }, + "source": [ + "#### Load a (frozen) TensorFlow model into memory" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "xHau6ysLi2yv", + "colab_type": "code", + "colab": {} + }, + "source": [ + "detection_graph = tf.Graph()\n", + "with detection_graph.as_default():\n", + " od_graph_def = tf.GraphDef()\n", + " with tf.gfile.GFile(PATH_TO_FROZEN_GRAPH, 'rb') as fid:\n", + " serialized_graph = fid.read()\n", + " od_graph_def.ParseFromString(serialized_graph)\n", + " tf.import_graph_def(od_graph_def, name='')" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b0VnLMMvmtPp", + "colab_type": "text" + }, + "source": [ + "#### Loading label map\n", + "Label maps map indices to category names, so that when our convolution network predicts `5`, we know that this corresponds to \"airplane\". Here we use internal utility functions, but anything that returns a dictionary mapping integers to appropriate string labels would be fine" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ZofMI3VLi-Vz", + "colab_type": "code", + "colab": {} + }, + "source": [ + "category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)\n", + "print(category_index)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AMwE_dYXmxPY", + "colab_type": "text" + }, + "source": [ + "#### Helper code" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qVvBib_QjDl9", + "colab_type": "code", + "colab": {} + }, + "source": [ + "def load_image_into_numpy_array(image):\n", + " (im_width, im_height) = image.size\n", + " return np.array(image.getdata()).reshape(\n", + " (im_height, im_width, 3)).astype(np.uint8)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Xdt4jDREiqn2", + "colab_type": "text" + }, + "source": [ + "#### Inference function\n", + "Perform inference one image at a time" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "TYJ5UrIMiuh2", + "colab_type": "code", + "colab": {} + }, + "source": [ + "def run_inference_for_single_image(image, graph):\n", + " with graph.as_default():\n", + " with tf.Session() as sess:\n", + " # Get handles to input and output tensors\n", + " ops = tf.get_default_graph().get_operations()\n", + " all_tensor_names = {output.name for op in ops for output in op.outputs}\n", + " tensor_dict = {}\n", + " for key in [\n", + " 'num_detections', 'detection_boxes', 'detection_scores',\n", + " 'detection_classes', 'detection_masks'\n", + " ]:\n", + " tensor_name = key + ':0'\n", + " if tensor_name in all_tensor_names:\n", + " tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(\n", + " tensor_name)\n", + " if 'detection_masks' in tensor_dict:\n", + " # The following processing is only for single image\n", + " detection_boxes = tf.squeeze(tensor_dict['detection_boxes'], [0])\n", + " detection_masks = tf.squeeze(tensor_dict['detection_masks'], [0])\n", + " # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.\n", + " real_num_detection = tf.cast(tensor_dict['num_detections'][0], tf.int32)\n", + " detection_boxes = tf.slice(detection_boxes, [0, 0], [real_num_detection, -1])\n", + " detection_masks = tf.slice(detection_masks, [0, 0, 0], [real_num_detection, -1, -1])\n", + " detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n", + " detection_masks, detection_boxes, image.shape[0], image.shape[1])\n", + " detection_masks_reframed = tf.cast(\n", + " tf.greater(detection_masks_reframed, 0.5), tf.uint8)\n", + " # Follow the convention by adding back the batch dimension\n", + " tensor_dict['detection_masks'] = tf.expand_dims(\n", + " detection_masks_reframed, 0)\n", + " image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')\n", + "\n", + " # Run inference\n", + " output_dict = sess.run(tensor_dict,\n", + " feed_dict={image_tensor: np.expand_dims(image, 0)})\n", + "\n", + " # all outputs are float32 numpy arrays, so convert types as appropriate\n", + " output_dict['num_detections'] = int(output_dict['num_detections'][0])\n", + " output_dict['detection_classes'] = output_dict[\n", + " 'detection_classes'][0].astype(np.uint8)\n", + " output_dict['detection_boxes'] = output_dict['detection_boxes'][0]\n", + " output_dict['detection_scores'] = output_dict['detection_scores'][0]\n", + " if 'detection_masks' in output_dict:\n", + " output_dict['detection_masks'] = output_dict['detection_masks'][0]\n", + " return output_dict" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RJK8_5ixe3a8", + "colab_type": "text" + }, + "source": [ + "### Detection - Validation Images\n", + "Detect wind turbines in validation image set. \n", + "\n", + "This section provides the validation images containing at least one wind turbine and corresponding bounding box(es)." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ImjGe0yPe6lC", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import os\n", + "import glob\n", + "\n", + "PATH_TO_TEST_IMAGES_DIR = '/content/wind-turbine-detector/images/valid'\n", + "TEST_IMAGE_PATHS = glob.glob(os.path.join(PATH_TO_TEST_IMAGES_DIR, \"*.*\"))\n", + "print(TEST_IMAGE_PATHS)\n", + "\n", + "# Size, in inches, of the output images.\n", + "IMAGE_SIZE = (6, 6)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "odIH7bYjfe-s", + "colab_type": "text" + }, + "source": [ + "#### Perform inference on all validation images\n", + "Loop over validation set and output images containing detected wind turbine with its location denoted with a bounding box." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "yd7jvqMxfApD", + "colab_type": "code", + "colab": {} + }, + "source": [ + "for image_path in TEST_IMAGE_PATHS:\n", + " image = Image.open(image_path)\n", + " # the array based representation of the image will be used later in order to prepare the\n", + " # result image with boxes and labels on it.\n", + " image_np = load_image_into_numpy_array(image)\n", + " # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n", + " image_np_expanded = np.expand_dims(image_np, axis=0)\n", + " # Actual detection.\n", + " output_dict = run_inference_for_single_image(image_np, detection_graph)\n", + " # Visualization of the results of a detection.\n", + " vis_util.visualize_boxes_and_labels_on_image_array(\n", + " image_np,\n", + " output_dict['detection_boxes'],\n", + " output_dict['detection_classes'],\n", + " output_dict['detection_scores'],\n", + " category_index,\n", + " instance_masks=output_dict.get('detection_masks'),\n", + " use_normalized_coordinates=True,\n", + " line_thickness=8)\n", + " plt.figure(figsize=IMAGE_SIZE)\n", + " plt.axis('off')\n", + " plt.imshow(image_np)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mqU0ar1Jm58Q", + "colab_type": "text" + }, + "source": [ + "### Detection - Full NAIP Images\n", + "Detect wind turbines in full size NAIP images (~4 miles square) using sliding window of size 300 x 300 pixels over the entire 5978 x 7648 pixel image. \n", + "\n", + "This section outputs the individual images containing at least one wind turbine and corresponding bounding box(es), the full NAIP image with all the wind turbines marked, and the latitude and longitude coordinates of all detected wind turbines. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "WcYD09DjjR_1", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import os\n", + "import glob\n", + "\n", + "PATH_TO_TEST_IMAGES_DIR = '/content/wind-turbine-detector/images/samples/orig'\n", + "TEST_IMAGE_PATHS = glob.glob(os.path.join(PATH_TO_TEST_IMAGES_DIR, \"*.jpg\"))\n", + "print(TEST_IMAGE_PATHS)\n", + "\n", + "# Size, in inches, of the output images.\n", + "IMAGE_SIZE = (8, 8)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EwAfAVqLgDpS", + "colab_type": "text" + }, + "source": [ + "#### Perform inference on full NAIP images\n", + "For each full NAIP image, perform inference on sliding 300 x 300 pixel window. Output images containing detected wind turbine with its location denoted with a bounding box." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ZgbO1Ov8j7rm", + "colab_type": "code", + "colab": {} + }, + "source": [ + "BBsWTs = np.empty((0, 4))\n", + "detectedWTs=[]\n", + "chipsize = 300\n", + "for image_path in TEST_IMAGE_PATHS:\n", + " image = Image.open(image_path)\n", + " width, height = image.size\n", + " ii=0\n", + " for x0 in range(0, width, chipsize): # width\n", + " ii+=1\n", + " jj=0\n", + " for y0 in range(0, height, chipsize): # height\n", + " jj+=1\n", + " box = (x0, y0,\n", + " x0+chipsize,\n", + " y0+chipsize)\n", + " # the array based representation of the image will be used later in order to prepare the\n", + " # result image with boxes and labels on it.\n", + " image_np = load_image_into_numpy_array(image.crop(box))\n", + " # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n", + " image_np_expanded = np.expand_dims(image_np, axis=0)\n", + " # Actual detection.\n", + " output_dict = run_inference_for_single_image(image_np, detection_graph)\n", + " BBsWTs=np.append(BBsWTs,output_dict['detection_boxes'][output_dict['detection_scores']>0.5],axis=0)\n", + " if len(output_dict['detection_scores'][output_dict['detection_scores']>0.5])>0:\n", + " # Visualization of the results of a detection.\n", + " vis_util.visualize_boxes_and_labels_on_image_array(\n", + " image_np,\n", + " output_dict['detection_boxes'],\n", + " output_dict['detection_classes'],\n", + " output_dict['detection_scores'],\n", + " category_index,\n", + " instance_masks=output_dict.get('detection_masks'),\n", + " use_normalized_coordinates=True,\n", + " line_thickness=8)\n", + " plt.figure(figsize=IMAGE_SIZE)\n", + " plt.axis('off')\n", + " for kk in range(len(output_dict['detection_scores'][output_dict['detection_scores']>0.5])):\n", + " plt.plot(chipsize*np.mean([BBsWTs[-1-kk][1],BBsWTs[-1-kk][3]]),chipsize*np.mean([[BBsWTs[-1-kk][0],BBsWTs[-1-kk][2]]]),'bo')\n", + " detectedWTs.append([image_path,ii,jj])\n", + " plt.imshow(image_np)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "SlQByra45P0B", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# total number of wind turbines detected \n", + "print(len(detectedWTs))\n", + "print(detectedWTs)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "h0krBKFEgiK8", + "colab_type": "text" + }, + "source": [ + "####Bounding box center location\n", + "Calculate normalized coordinates of bounding box center. This will be used to plot each detected wind turbine on the NAIP images and to compute the latitude and longitude of each wind turbine." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "JYIabjFp8GLq", + "colab_type": "code", + "colab": {} + }, + "source": [ + "centerBBs = np.empty((len(BBsWTs),2))\n", + "ii=0\n", + "for BBs in BBsWTs:\n", + " centerBBs[ii][:]=[np.mean([BBs[1],BBs[3]]),np.mean([[BBs[0],BBs[2]]])]\n", + " ii+=1" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LK-8KtshcUZZ", + "colab_type": "text" + }, + "source": [ + "####Plot wind turbine locations over NAIP image\n", + "Plot marker for each detected wind turbine on NAIP images analyzed." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "XlPLn9Mtjpgn", + "colab_type": "code", + "colab": {} + }, + "source": [ + "for image_path in TEST_IMAGE_PATHS:\n", + " image = Image.open(image_path)\n", + " plt.figure(figsize=(width/500,height/500))\n", + " plt.axis('off')\n", + " for ll in range(len(detectedWTs)):\n", + " if detectedWTs[ll][0]==image_path:\n", + " x_plot=chipsize*(detectedWTs[ll][1]-1+centerBBs[ll][0])\n", + " y_plot=chipsize*(detectedWTs[ll][2]-1+centerBBs[ll][1])\n", + " plt.plot(x_plot,y_plot,'ro')\n", + " plt.imshow(image)\n" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "InaVNI0_bVxj", + "colab_type": "text" + }, + "source": [ + "####Calculate wind turbine latitude and longitude\n", + "Perform a series of 1D linear interpolations to determine latitude and longitude coordinates of each detected wind turbine." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "pc0NwxvFqst2", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# initialize dictionaries for NAIP image corners\n", + "# y1,x1 are lat,long of upper left corner of image\n", + "# y2,x2 are lat,long of upper right corner of image\n", + "# y3,x3 are lat,long of lower right corner of image\n", + "# y4,x4 are lat,long of lower left corner of image\n", + "x1={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':-94.8178888,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':-94.8180499}\n", + "y1={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':41.3151166,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':42.6276193}\n", + "x2={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':-94.7464999,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':-94.7464221}\n", + "y2={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':41.3162222,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':42.6287332}\n", + "x3={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':-94.7446638,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':-94.7444999}\n", + "y3={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':41.2473638,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':42.5598722}\n", + "x4={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':-94.8159777,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':-94.8160472}\n", + "y4={'/content/wind-turbine-detector/images/samples/orig/m_4109442_se_15_1_20170709.jpg':41.246261,\n", + " '/content/wind-turbine-detector/images/samples/orig/m_4209426_ne_15_1_20170707.jpg':42.5587611}\n", + "\n", + "# loop over all the detected wind turbines and calculate the \n", + "# latitude and longitude coordinates based on its location in\n", + "# full NAIP images with respect to the image corners\n", + "latlongWTs=[]\n", + "for ll in range(len(detectedWTs)):\n", + " a5=chipsize*(detectedWTs[ll][1]-1+centerBBs[ll][0])\n", + " b5=chipsize*(detectedWTs[ll][2]-1+centerBBs[ll][1])\n", + "\n", + " p_a5=a5/width\n", + " p_b5=b5/height\n", + "\n", + " x_le=x1[detectedWTs[ll][0]]+p_b5*(x4[detectedWTs[ll][0]]-x1[detectedWTs[ll][0]])\n", + " x_re=x2[detectedWTs[ll][0]]+p_b5*(x3[detectedWTs[ll][0]]-x2[detectedWTs[ll][0]])\n", + " x_WT=x_le+p_a5*(x_re-x_le)\n", + "\n", + " y_te=y1[detectedWTs[ll][0]]+p_a5*(y2[detectedWTs[ll][0]]-y1[detectedWTs[ll][0]])\n", + " y_be=y4[detectedWTs[ll][0]]+p_a5*(y3[detectedWTs[ll][0]]-y4[detectedWTs[ll][0]])\n", + " y_WT=y_te+p_b5*(y_be-y_te)\n", + " latlongWTs.append([y_WT,x_WT])\n", + "\n", + "# print out latitude and longitude coordinates for verification or plotting\n", + "print(latlongWTs)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "u8QmWQhTrnJO", + "colab_type": "text" + }, + "source": [ + "## Summary\n", + "\n", + "The trained model accurately detects at least 15 out of 17 wind turbines in the validation image set with high probability. This represents an accuracy of ~90%. Higher accuracy would likely be achieved by using a larger set of images (train + test) as well as using a more accurate pre-trained model. Alternative models, including those with higher mAP, can be found at the [TensorFlow Object Detection Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md). Details of the trade-offs between speed, accuracy, and memory for various object detection model architectures (Faster RCNN, SSD, R-FCN) can be found in this [paper](https://arxiv.org/pdf/1611.10012.pdf), which can serve as a good starting point in determining which architecture is best for your application.\n", + "\n", + "Using the trained model, images encompassing a large area (~16 sq miles) were scanned for wind turbines. Once detected, the position of each wind turbine was plotted on the original full NAIP image and its position (latitude, longitude) was output. The comprehensive pipeline including training, validation, and inference could be applied to many other applications involving detection of objects from aerial or satellite images. " + ] + } + ] +} \ No newline at end of file