{ "cells": [ { "cell_type": "markdown", "metadata": { "toc": "true" }, "source": [ "# Table of Contents\n", "

1  Lab Overview
1.1  Important things to note about Python/Jupyter
1.2  How to load the Jupyter notebook
1.3  Learning Outcomes
2  Background
2.1  Introduction
3  Data Exploration
3.1  Set Up Libraries
3.2  Import Data
3.3  Displaying the data
3.4  Correlation Matrix
4  Building a Machine Learning Model
4.1  Linear Regression
4.2  Evaluating Model Performance - Coefficient of Determination ($r^2$ value)
4.3  Over- and Underfitting
4.4  Splitting Data into Training and Testing Sets
4.5  Building the Model
4.5.1  Training
4.5.2  Testing
4.5.3  Visualising the Predictions
5  Feature Selection & Cross Validation
5.1  Polynomial Regression
5.2  Cross Validation
5.3  Training and Testing Model on Best Feature Set
6  Making future predictions
6.1  Input into existing model
6.2  Compare Predictions against Housing Dataset 1
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab Overview\n", "## Important things to note about Python/Jupyter\n", "Welcome to Jupyter. This is a _notebook_ development environment for python. If you've used mathematica before, you'll be familiar with the notebook concept.\n", "\n", "Each cell (like this one) can either be for `code` or _markdown_ (a very basic language for formatting text). This cell is a markdown cell.\n", "\n", "Clicking in the `typing` area of a code cell will let you enter edit mode on that cell. Pressing ctrl+Enter will run the code in that cell. Keyboard shortcuts can be found by clicking on help -> keyboard shortcuts.\n", "\n", "Please note that any variable created in the code can be viewed by typing `print(variable)`.\n", "\n", "\n", "The point of this exercise is to introduce you to some of the basic concepts of machine learning and data analytics, and apply knowledge of what you've learned in CE3010/NE3002/NE6012 to a real-world problem. We use python for this exercise as it is ubiquitous in the world of data science and machine learning, and is good to get some exposure to. However, you will not be graded on your understanding of python or programming in general. You should be able to gain an intuition of what the code is doing by reading through, and all the code is annotated to help you understand what's going on.\n", "\n", "## How to load the Jupyter notebook\n", "1. Open **Chrome** and go to https://notebooks.azure.com\n", "2. Sign in with your UCC IT account. If you can't remember this, but have a microsoft/hotmail account, you can sign in with this. Otherwise, you'll need to create an account. **Don't forget the password if you're creating an account**.\n", "3. Click on \"Libraries\" in the top left\n", "4. Click on \"New Library\", then click on \"From GitHub\"\n", "5. The GitHub Repo is https://github.com/lkev/ce3010_lab\n", "6. Give it the name \"CE3010 Lab FirstName LastName\" (with your actual name)\n", "7. Give it the ID ce3010-firstname-lastname\n", "8. Click Import\n", "9. If the import seems to be taking too long, refresh the page.\n", "10. In the library folder, click on \"Housing - Partially Filled\". This is the notebook we'll be using for this lab.\n", "\n", "The lab will be structured as follows:\n", "\n", "* Lab instructor will go through this jupyter notebook (Housing - Partially Filled), giving a background to machine learning and taking you through a machine learning problem with the `household_data` dataset\n", "* You will then be asked to perform a similar analysis on the `boole_data` dataset, and write up the results (details found in the Boole - Partially Filled notebook).\n", "\n", "You are allowed to copy and paste or otherwise edit some of the code here to apply it to your own datasets if you are not comfortable writing from scratch. As well as this, 99% of errors or anything else you are having trouble with can be solved by a quick google. In particular, the the [cross-validated](http://stats.stackexchange.com/) and [stack-overflow](http://stackoverflow.com/) stack exchange sites are great resources." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Learning Outcomes\n", "Data analytics allows implicit, previously unknown, and potentially useful information to be extracted from data. It is an interdisciplinary subfield of computer science that combines artificial intelligence, statistics, machine learning and database research. As the volume of data increases, the proportion of it that people understand decreases significantly. Lying hidden in this data is information that is potentially important but has not yet been discovered or articulated. As a result, data analytics allows the useful information within this data to be successfully accessed and analysed. Data analytics and machine learning have many applications in modern engineering sectors, including:\n", "* Building & Energy Analytics\n", "* Industrial Manufacturing\n", "* Engineering Design\n", "* Predictive Maintenance\n", "* Fault Detection & Performance Monitoring\n", "* Self-driving cars\n", "* Banking\n", "* Phone typing\n", "* Image recognition\n", "\n", "**The objectives of this assignment include the following:**\n", "* To gain an understanding of the concept of data analytics and its application in buildings for energy performance analysis.\n", "* Get introduced basic machine learning principles and the importance of having a separate test and training set\n", "* Understand the effect of daylight hours, occupancy, heating degree days (HDD) and building opening hours on electrical energy consumption in buildings.\n", "* An introduction to data analysis using python, and the sklearn, numpy and pandas libraries\n", "* To investigate and analyse the energy performance of a UCC building using both correlation and regression analysis\n", "* To predict the future electrical energy performance of the building using the developed regression model\n" ] }, { "attachments": { "data_matrix.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAApUAAALDCAMAAABzWAt/AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAMAUExURQAAABERESMjIzQ0NEdHR1dXV2lpaXp6en9/fzFllDZpljlrmDttmT5vmkBvmUBvmkBwmkBwm0FxnEJynUJynkJzn0N0oEN1oUR2o0V3pEZ4pUV4pkZ5p0d6p0Z6qEd6qUd7qk57o0h7qkh8qkh8q0h9rUl+rUl+rkp/r0qAsEqAsUqAskyCtE2DtU2Etk2Ft02FuE6GuU+Huk+Hu0+Iu0+IvFWAp1eBqFCIu1CIvFCJvVCKvVGKvlKMwVONwlOOw1OPxFOOxVSQx1WRyFWSyVaTylaUzFeVzViWzliWz1iX0FmY0VmY0lqZ01qa1Fua1Vub1luc1Vuc1lud11yc1lyc11yd2Fye2Fye2V2e2l2f24UAEoYAE4cAFIgAFYkBFooCF44AFokDGIsEGIwFGY0GGo4IG48KHJUAF5ADGZAKHZEMHpINH5MPIJMQIJQQIZYTI5YTJJsaKZwbKqIjMKIkMKMlMagqNqksN60yO640PbQ7QrQ8QrU9RLY+RbhBR7hCR7pESb5JTb5KTb9KTrdlcrprd7ttecBLT8BMT8FMUMFOUMJOUcNPUsNQUsNQU8RRU4mJiZiYmKenp6qqqrW1tZu0y5+3zJ+3zZ+4zaC4zaC4zqC5z6G60KG60aK80qK806O91Ka80Km/0qq/08aDjciIksiJk+G/xOO/xcPDw8vLy9DQ0N3d3c3Z5c3a5c7b5tDb5tTf6e7a3eDg4Ozs7Obs8unu8/Hh5Pfu7/ju7/L1+PL2+PP2+fT3+fr19vv3+Pj6+/z9/f///wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI4M8WoAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAZdEVYdFNvZnR3YXJlAHBhaW50Lm5ldCA0LjAuMTczbp9jAAAi5ElEQVR4Xu3diZtj2XmQ8Xu1VIWYzEz1JI7NYgIZMPGEgAlLDAYSSIxjG9uEzKTTdndsbA8JS8y+Q1jCYmcAq0TjYQthJ6IEYRFLbJZQ/xjf+c53Tt1PdUpSaTn3pvt9n3mqpKuSdOfUr85dVF1qromGFippeKGShhcqaXihkoYXKml4oXLfJk1qJtfO7fIk3kgHhcp9m7cG8Xwl156O9fL4abyRDgqVe/dUHabZcREun8XLdGCo3L+4DTeVs5uLdGio3L+5qhzp5ZVsz8dhU05HCJUHFPcsl+GizJstKI8VKg8oHniHQ3DZfrdXceFzmu5XS50r++/QoPKArnTwZRMevgtzW/ictjzTwYielvHnde+NByoPKZ4OWoSdyjBjPt/Fgz/bYoSRYa7sp0v9RpzJt4BzQjYYi3glqNx/64HKQ1rqN0Li8DvtTJpKuRRPTuwVKg9qqt8JDr9DcS877skcuKONyoOKpyyf88PvlI5FVClHOwdMlag8sM4py+c+HYuoUi4eck4ClQe1GqnKS7v6fKcH4XrgLZuQQ6ZKVB6WnaQ76FvwzHSjUn5W7ahnv1B5SOm3KtNZuue78AsqqlKmyv3PVYZQeUAy+m2cLTldKalKARV2aw7b00bl/oVfsJzHU5atLXqui79wqjoPfKULlXt3FV9njKcsn/NXwbV4Gv162TajA8/fonLfwovfYcMdT1lO48LnurjZWMiP6UGHOhIq92w1ttcZg06JU5aCKSRHgOd2fe9QuWc3v+Ybj3c4ZaknhEKHv/6Kyv0KFG07FXfyOWWZfpft4O03KvcsnARJu5LxtxKO8L34eV888Dt4+43K/dIjnHT2w36djV9miycsDz3+DqHy/l3FOaGNu5L29wmE5XP/Ao+qPMYooPL+xdcwhKFes118ib+bIcffRznqQyUdLTnuO855W1TSsbo6/EUdC5V0pFbjo/1SPirp0Gb6e2vhxa5j7Vmjkg5tEg68n8pR39F+RQWVdGjxFwGO+XtTqKQDs1cR2iO+uIVKOrCr8PL35PKYL22hkoYXKml4oZKGFyppeKGShhcqaXihkoYXKml4oZKGFyppeKGShhcq6e7iP5SbVP8LDKiku2ua8WQySv9url6opLuLf3thUf1PzqGS7s7+IshZ7T8515fK9E/7j/xDeNnu/j90ilVYnsujjg/8m6KpE6yg/gG5SfpVyPDHkjb+q0RTGf9hTvjzDG3Tnulfnwt/qaGdyn2vROxs1LTn8XGuzuQZJrrGy2a6kltGcTTyHcoP4+tLpe6xSJt+gXl637/dswy/f2qXt3eCVQj/9jTshx3nD1KfYAXjHz61o5f4Z7s23d9uHcd7zMXebKr/kPEqXDwbyaVFMx7Letq+Z3jziIn8NIUBiLfIlfCHh27uUH4YX38qt4/m5H4jfj1vm9m9VB59FRYqSKag+634HZ1iBUWPAQqiw8y56f5661K+Lsxmq7YN89s8TJzT/M8Z5RFbuSw/jzJBLlv1u9Ar4Ra5u0CVe9/cofwwvmdJ5WR8JY9rV7Z3ilWIyU+HXTqo06icyEZTt54rmdK2qtQm+vWXNseO5O6Trkqd6s7Dvuelbep1k59uGYWnuLlD+WF8g1DZ2dFY6Y5Z2C+xv+Yzk2k+/q9Ow/9k2FmZxH8N37lXLDziviqPtAqxyQlUHmcFg0qZvPSP+clnubpF5Vh2SMJcKE2as1koKJs1I7ufPIR9HoUvCWslTxsWplv0B+fmDuWH8Q1BZWdHI/yl97BfIt/V+aQN+1Vz+b+LI67/d7qzMolbhnyvTnuqPOYqhLnS9twO6wQrGFTKHKl/AnYsn7aq1AeJc6v9zVRJHk4m3JH+A7IblfK5TY92S+XNHcoP49v9m3jcOqPR3dFYjMPiy/hOIfF/R/7v3IjbGHfv1Wk/lUddhTBTZAaHdIIVlJsmgYJ81TJMmWIjP0eheOtZPHqzp7LCH+UPf7Iz2dPP+dFuq8x3KD+Mb/dv4nEL24aQXOzuaFi6g3zXiMftSeFeoXupPM0qyFdlBQd1ghWUmyaiNDA7Dz87O6kUv+H+afOcWo7DwyR7T8MxVNpPlGPrm1tsFdMdyg/j609lTC52dzSs+D9yx4jrldK9QvdSGZOLR10F2Wx6pftm63fMFZSbJuEwR9CMwnZcHj0/YCG7daz7yef2HDl9UnlEnZd1QrVZVfYX5dAnrYatoqR3KD+Mz9a/ep3RcDsa16uneRi3jHjKbS3TzTt0olW4au84sLx3J1hBuUm+ULaa86d6zCM35ecoZLfOdUdUpkydeudyPz0AOre5MuzByif5KPsu4UvksTuroet0c4fyw/h2/yYeNzfi+WJYxVhYtGXEO/e6aV+VnQc7bBXiibujdIIVlJvkC2UTPj1TqvYod2W3ruIPmtytnYh0mQfjqwVh71UeMZ4rV1qzcBbd/g5WWg1di5s7lB/Gt/s38bi5Eb/5Ns6acfj/j8OZRzye9V0f8eI3f1+Vx1oF+Qbe+snftxOsoNwkKmUTLoX5b4vK1ibZs3guaTENLyeGY+blWTjjFG6UR1zJlVHcg71+KvNwO9XHXMbjMZmZ5erNHcoP4xuAyu6Ohg1yGnEdVBnDsF+/0k1WHvFbuyex/VQebRVWhV33vTvBCspNYZHQiSctt6jcofxkx+wED7lTndHo7mjEgziZ48OtM/sOj8IAXsl2oTvi3Xt1ymM0jTdv6ASr4FHesQqjMAHNwwZMLoYniR+vr//+1/x7/Zw6wQqaSrlz3NdEpas7Gp0djctw2ncieyrhVvkfHoffv7kMp0hku+BGvHsvazYJ1+VDGO/tr6+cYBVkt0rP5Uz0NbryKsjdZbHcK12MH6X1rz/BCspNQWXYhOuZdFS60h6LdrOjcT2X/ebpYhZvnY+aUZgXLmWP+GylC5dxNEOde8V0uxQKA50moLs7wSqIDSvcubwKqzbs3c91jvNz5frXn2AF5SBZhcqxjv4EHP6nzNPe41HrS+Wpi9/1XrvvKgxglYfSs6pydKT32Dig+67CAFZ5KD2jKudbt98n776rsPya/2qX6FmdK+nnc6ik4YVKGl6opOGFShpeqKThhUoaXqik4YVKGl49qPxf//FnSPrvNh65/2Y39Nx/jmvzP37639Xsp/9LfFqtB5X/4Re8jd72tq/9FzYeuX9gt/Tb1/7yuDb/+oc/XbMf/hvxabUeVP7M2x7Qgwdf9y9tPHJv2S399nW/Ka7Nv/n0k5p9+m/Gp9VQ2VeodKFyEKHShcpBhEoXKgcRKl1DUXnRV/Hp+1uB+Ox3q7Qvq1589nWVj09dfJqBqLz47u/9YC994F0vhed/4d0ftgWV+/C3vRCe/7bKnxrGejmVjz/3V37stP3VP/zJIan80O/7vl766DfH7/57HtqCyj389jtU/vNhrJdX+cYX/vZpe/OPDErl937fx3rp9ySVr9uCyr3+67eo7Hm91lT++In7IipDqCyHSgmVNh45VKZQWT1UulCpobIcKiVU2njkUJlCZfVQ6UKlhspyqJRQaeORQ2UKldVDpQuVGirLoVJCpY1HDpUpVFYPlS5Uaqgsh0oJlTYeOVSmUFk9VLpQqaGyHColVNp45FCZQmX1UOlCpYbKcs+Uys77s+0WKlHZaYvK2bhpxsV3ad3YhreXvGxLN6ESlZ02qxST4e1R7/1egneqXMqjofJ2qHRtVHnZjJbX16tZftPeXbtL5bxtZqgshErXRpXb30/7ju5SORlfXaOyECpdu6tcnYd9zHm4uGymq1mr70E9GzUj3e+8aqbhSnse3qQ1qbyatk17Ft6tPRbeWxOVhVDp2qhy1nn/8qUonIjL4HTRjMfNeNQ080m41ASWcZnsNo71ij7GPLzD/3TtTdBRWQiVro0qVwIvzXSLcZjoLvVd8gVdK9emcuQic6fsfdoymTuv2rAoqly1bbjzvJnIxxwqC62rHKW3bEZl6obNKhwyd7bAwZRsoQVdmP6WitIvuz6XLbmpvNRJNAyxfwS70A2VTqWMn+07oTLVZbMQl63ai03CD3FEl4HpD3ZatggzZ7wyac5mofyDr6GyEHOla5tKESY7jjrXrZ4mYWsqnVT9nFSmujuW9nU+VHqVOVSm1tjI3mXYnJybsPuotB94l32dD5Wo7LSDSvEoKmfNWA5mvED75JY9DQfhSWVn059DZSFUunZROQ5HLTbxbVKp2+mz8Ppk/IJzf/BtobIQKl2bVC5H83BK/FL3KydNmCrnm7bg4WyRfMpfIMfoehA+Pw8fU6gshErXJpWrNvxyhnwIG+LLcEZ80ow3qBzrKfXwqxz2BWK4Db/dkV9Hn03CVfngzqtLqERlp41b8GV4kXF0FgnN5fJ0MQvb6aWeS7++buOnSVgmEFdnbX75Md6ymAbYl2HG1cJ5d00Yu1CJyk677FfuVJo/9wuVqOyESg2V5VApodLGI4fK1F680r7mfqESlZ2OpvKwUInKTqjUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKqLTxyKEyhcrqodKFSg2V5VApodLGI4fKFCqrh0oXKjVUlkOlhEobjxwqU6isHipdqNRQWQ6VEiptPHKoTKGyeqh0oVJDZTlUSqi08cihMoXK6qHShUoNleVQKaHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sZjlN7XH5UpVFZvTeWiaWbxEipTqKwec6ULlRr7leVQKaHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKFx/8vR/tpY8kla/Zgsq9tk1lz+u1pvKv/7XT9oVBqXzwDW/vqZfj879sV6v39fr0t1X+Q13e+3o5lU+efPbU/ZA+zVBUPrjoKXv6np//tsq34g19r9eaysenLj7NYFQ+392tst/WVVYKlYMIlS5UDiJUulA5iFDpGorKF/rKdusv7Gr19ARQQeWXdXnv67Wm8hOnLh7uDETlxXt/62/ppd/4TmX50rvebwsq9/5fpd/+2yp/chjr5VQ+/syfOXF/9g1lORSVH3n4ei99/JV4tvrVR7agco/et/kset/r5VW+8eaJ+9LnecVR4hXHcrziKKHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHStdpVC6bkV3aMVSistMWlbNx04wv7cruLZqy7OV5eLyZXeuESlR22qxSDE0mTXNmV3fuDpVXbTOajEqPh0pUdtqo8rIZLa+vV7OpXd+5O1QuJgv5eNY04ZMLlajstFHlpClsbHfpri14rL39sKhEZafdVa50n3AeLi6b6WomW+OnsuM5aka633nVTMOV9nwlV5LKq2nbtGcy37oK2FGJyk4bVc6a9souXi/DPqG4DKAWzXjcjGUHcT4Jl5rAMi6TndCxXtHHmAvS2bTzILFW7+BCJSo7bVS5CkcmNtMtxmFn8LJpw+WmaeXatBGXuvdpy2TulAMaWRRVrto23HneTOTjTVdNs8YUlesqR2nXG5WprPJ6JXNfdqk1jWyhBV1wtVSUftn1uWzJTeWlzYmjxm3DJ2tKQ6h0KmX8bCcHlakblTI+4rJVe7FJ+CGO6AJH/aQ/2GnZIsyc8cqkOZuF8g++Jpv19R1NVDJXurapFGGy46iMVk+TsDWVTqp+TipTnS22bOJlQ78eKr3KHCpTkVdO9i7D5uTchN1Hpf3Ad4r7nbdCJSo77aBSPIrKWTMOc1xpC+6WPQ0H4UnlLYCrtvxCESpR2WkXleNw1GIT3yaVup0+C+ziF5zfOqxZje949RKVqOy0SeVyNA+nxC91v3Kiu4PzTVvwcLZIPuUvkGN0PQifn4eP0p0oUYnKbptUyua2mUzkQ9gQX4Yz4pNmvEHlWE+pB3f2BWK4Db/dkV5Hn4XH09aPwlGJyk4bt+D6i2ejs3gEPZfL08UsbKeXei79+rqNnyZhmUBcnbX55cd4y2IaYF+GGTcks67VOSjXUInKTrvsV+5Umj/3C5Wo7IRKDZXlUCmh0sYjh8rUXrzSvuZ+oRKVnY6m8rBQicpOqNRQWQ6VEiptPHKoTKGyeqh0oVJDZTlUSqi08cihMoXK6qHShUoNleVQKaHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKqLTxyKEyhcrqodKFSg2V5VApodLGI4fKFCqrh0oXKjVUlkOlhEobjxwqU6isHipdqNRQWQ6VEiptPHKoTKGyeqh0oVJDZTlUSqi08cihMoXK6qHShUoNleVQKaHSxmPULOIFVKZQWb01lYummcVLqEyhsnrMlS5UauxXlkOlhEobjxwqU6isHipdqNRQWe5ZUZn30+8VKlHZ6dgqb85p3CtUorITc6WGynLPisr9QiUqO6FSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKqLTxyKEyhcrqodKFSg2V5VApodLGI4fKFCqrh0oXKjVUlkOlhEobjxwqUz2q/NBr399LH00qH9qCyj38DVtU9rxeayq/cOLeHJbK3/m7P9BL3/Ou+N1/9wdtQeU++GvuUPlTw1gvr/Jzf/nE/dgfGpLKBxcv9dNFfPrent9W4LbKt3R57+vlVD558slTF59mKCqf8+5U2XPrKiuFykGEShcqBxEqXagcRKh0oXIQodI1FJUvvWjHfrWLT//goqfnf3HbMXjP67Wm0o6UT1d8moGovPiu7/nuXvpdv1RdvvgrP2ALKveBb30xPP9tlf9sGOvlVD7+3F/8C6ftLw3qfOXFh15/rZc+Zq/tvPqDtqByP7jltZ2+18urfOOLJ+7v8Dp4iNfBy/E6uIRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1IamMr/LOipTqKzemspF08ziJVSmUFk95koXKjX2K8uhUkKljUcOlSlUVg+VLlRqqCyHSgmVNh45VKYOUpmPHu8VKlHZ6dgqb8603StUorITc6WGynLPisr9QiUqO6FSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKqLTxyKEyhcrqodKFSg2V5VApodLGI4fKFCqrh0oXKjVUlkOlhEobjxwqU6isHipdqNRQWQ6VEiptPHKoTKGyeqh0oVJDZTlUSqi08cihMoXK6qHShUoNleVQKaHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm088rusozKFyuqtqVw0zSxeQmUKldVjrnShUmO/shwqJVTaeORQmUJl9VDpQqWGynLPisq8n36vUInKTsdWeXNO416hEpWdmCs1VJZ7VlTuFypR2QmVGirLoVJCpY1HDpUpVFYPlS5Uaqgsh0oJlTYeOVSmUFk9VLpQqaGyHColVNp45FCZQmX1UOlCpYbKcqiUUGnjkUNlCpXVQ6ULlRoqy6FSQqWNRw6VKVRWD5UuVGqoLIdKCZU2HjlUplBZPVS6UKmhshwqJVTaeORQmUJl9VDpQqWGynKolFBp45FDZQqV1UOlC5UaKsuhUkKljUcOlSlUVg+VLlRqqCyHSuniIw9f76WPvxK/+68+sgWVe/S+zSr7Xq81lW+euC99fkgqH7z7W/rpV3/jRXj6i3e8xxZU7j2/RJ//tsp/Moz1ciqffOqPn7g/8dnHQ1L54gv99KIOvnz77Xr14vPfVvllXd77enmVTz5x6uLTDEXlc95tlW/ZLf22rrJSqBxEqHShchCh0oXKQYRK11BUvtRX8ekfXLxo1ytnR1t3qux7vdZUfvLUxacZispf9so399I3vV2H/+Ibe3r+V96pz39bpZ0Z6nu9vMpP/dET98c+M6QzQxcf/oHXeulj+Sy6Lajco+/YdhbdvrByab2cysdv/MQXT9vf/VFecZR4xbEcrzhKqLTxyKEyhcrqodKFSg2V5VApodLGI4fKFCqrh0oXKjVUlkOlhEobjxwqU6isHipdqNRQWQ6VEiptPHKoTKGyeqh0oVJDZTlUSqi08cihMoXK6qHShUoNleWeFZWjZmGX7hMqUdnp2CoXTTOzi/cJlajsxFypobLcs6Jyv1CJyk6o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKqLTxyKEyhcrqravM59dQmUJl9dZU3rwWgcoUKqvHXOlCpcZ+ZTlUSqi08cihMoXK6qHShUoNleVQKaHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXajUUFkOlRIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHShcqNVSWQ6WEShuPHCpTqKweKl2o1FBZDpUSKm08cqhMobJ6qHShUkNlOVRKqLTxyKEyhcrqodKFSg2V5VApodLGI4fKFCqrh0oXKjVUlkOlhEobjxwqU6isHipdqNRQWQ6VEiptPHKoTKGyeqh0oVJDZTlUSqi08cihMoXK6qHShUoNleVQKaHSxiOHyhQqq4dKFyo1VJZDpYRKG48cKlOorB4qXUdXmd+t9V6hEpWdjq3y5p2t7xUqUdmJuVJDZblnReV+oRKVnVCpobIcKiVU2njkUJlCZfVQ6UKlhspyqJRQaeORQ2UKldVDpQuV2tBU5rO+qEyhsnprKm9eIUNlCpXVY650oVJjv7IcKiVU2njkUJlCZfVQ6UKlhspyqJRQaeORQ2UKldVDpQuVGirLoVJCpY1HDpUpVFYPlS5Uaqgsh0oJlTYeOVSmUFk9VLpQqaGyHColVNp45FCZQmX1UOlCpYbKcqiUUGnjkUNlCpXVQ6ULlRoqy6FSQqWNRw6VKVRWD5UuVGqoLIdKCZU2HjlUplBZPVS6UKmhshwqJVTaeORQmepR5Yde+/5e+mhS+dAWVO7ht29R2fN6ran8wol7c1AqH7zjnT319fH5v8GuVu/t+vS3Vf5jXd77ejmVT37oD566T+nzDEXlRV/Fp+97BW6rfEuX975eXuWTx6cuPs1QVD7n3a2y39ZVVgqVgwiVLlQOIlS6UDmIUOkaisoX+soOdy7savX0BFBB5Zd1ee/rtabyE6cuHu4MReW3/rpf20vf9g5lefGL32sLKvfeX6HPf1vlPx3GenmVn/lTf/K0/enPKcuBqLz4yMPXe+njr8Sz1a8+sgWVe/S+zWfR+14vp/LxG2+euC99nlccJV5xLMcrjhIqbTxyqEyhsnqodKFSQ2U5VEqotPHIoTKFyuqh0oVKDZXlUCmh0sYjh8oUKquHStfRVeZ3a71XqERlp2OrvHln63uFSlR2Yq7UUFnuWVG5X6hEZSdUaqgsh0oJlTYeOVSmUFk9VLpQqaGyHColVNp45FCZQmX11lXm82uoTKGyemsqb16LQGUKldVjrnShUmO/shwqJVTaeORQmUJl9VDpQqWGynKolFBp45FDZQqV1UOlC5UaKsuhUkKljUcOlSlUVg+VLlRqqCyHSgmVNh45VKZQWT1UulCpobIcKiVU2njkUJlCZfVQ6UKlhspyqJRQaeORQ2UKldVDpQuVGirLoVJCpY1HDpUpVFYPlS5Uaqgsh0oJlTYeOVSmUFk9VLpQqaGyHColVNp45FCZQmX1UOlCpYbKcqiUUGnjkUNlCpXVQ6ULlRoqy6FSQqWNRw6VqSOpXDYju7Q9VKKy0y4qZ+OmGV/ald1bNEXey7ZpRld6sW2m+hmVqHTtoFJMTiZNc2ZXd+4OlbJYHlAvps+oRKVru8rLZrS8vl7N0rS2c5tUNjpZojKFStd2lZO93lpZultlmnlRmUKl654qV+dhH3MeLi6b6WrWNqOnsuM5aka633klO4pypT1fyZWk8mraNu2ZzLcxWSxf0oTrqEyh0rVd5axp47GJJEcqo4m4DE4XzXjcjMXXfBIuNYFlXCZT4Viv6APNBelsevMgQeU8TpaoTKHStV3lSuClmW4xDm8Dc9m04XLTtHJtKjuJMnfK3qctk7nzqg2LospV24Y7zzNAWXx+HSdLVKZQ6dqu8nolc192qTWNbKFFV5j+lorSL7s+D6d8ospLnUTD+xol2cFinCxRmUKlaweVAklctmovNgnvnBXRBVr6Sd9NKy1bhJkzXpk0Z7NQfrctVSlf3q5QmUOlayeVQkl2HHWuWz1NwtZUOqn6OalM2Y5lVCmT5QyVOVS6dlQZ9i7DIc65CbuPSpsjU1FlmCxRmUOla1eV4lFUzpqxHMx4gSWVT8NBeFLZ2fSHTKVMlvIfKmOodO2schyOWmzi26RSt9Nn4VgmfsF5lmeZSpksw0uZugiVqHRtVbkczcMp8Uvdr5w0YaqUOe5uleFskXzKXyDH6HoQPj8PH6WkUh4FlSlUuraqXLViZyIfwob4MpwRn8gkd7fKsZ5SD+fI7QtEXxt+uyO9jp5Uht/6QKWFStf2LfgyvMg4OotH0HO5PF3MwnZ6qefSr6/b+GkSlom41VmbX36MtyymAfZlmHFDV+Esul4YpUuoRKVr5/3KnUrz571DJSo7oVJDZTlUSqi08cihMrW/yrSvee9QicpOx1W5d6hEZSdUaqgsh0oJlTYe+TesUJlCZfXWVMpRo/2zFFSmUFk95koXKjX2K8uhUkKljUcOlSlUVg+VLlRqqCyHSgmVNh45VKZQWT1UulCpobIcKiVU2njkUJlCZfVQ6UKlhspyqJRQaeORQ2UKldVDpQuVGirLoVJCpY1HDpUpVFYPlS5Uaqgsh0oJlTYeOVSmUFk9VLpQqaGyHColVNp45FCZQmX1UOlCpYbKcqiUUGnjkUNlCpXVQ6ULlRoqy6FSQqWNRw6VKVRWD5UuVGqoLIdKCZU2HjlUplBZPVS6UKmhshwqJVTaeORQmUJl9VDpGpjKD//Aa730sVfid//VR7agco++Y7PKvtdrTeVPfPG0/b0fHZTK9/+O395Lv+0XXYTnf+mbvtMWVO47v0X13Vb5k8NYL6/ys3/uxP35P/B4QCofvPhCT+k3X34s7Gr14vPfVvllXd77ejmVT5584tQpysGofM67rfItu6Xf1lVWCpWDCJUuVA4iVLpQOYhQ6ULlIEKlq2+V/+kXvkwvv3zxr2w8cv/Ibum3i98c1+bf/sjvr9mP/K34tFoPKv/fV75KX/3qV37OxiP3c3ZLv33lf8e1+b8/+z9r9rP/Jz6t1oNKoi2hkoYXKml4oZKG1vX1/wcukGwwEgLV/gAAAABJRU5ErkJggg==" } }, "cell_type": "markdown", "metadata": {}, "source": [ "# Background\n", "\n", "## Introduction\n", "\n", "Statistics, machine learning and data science can all be thought of as different sides of the same coin. The fields essentially boil down to different applications or uses for statistics, probability and, to some degree, information theory. As well as this, the different fields may have different terms for the same concepts. At the heart of all of these, however, is **data**. The datasets used in this study have been cleaned and prepared for you for easy manipulation.\n", "\n", "Each column in each dataset represents a number of **features** (also known as **independent variables** or **predictors**), as well as a column representing the **response** (also known as the **output** or **dependent variable**).\n", "Each row in the dataset then represents an individual entry, comprising the features and associated response for that sample. A diagram of this can be seen below:\n", "\n", "![data_matrix.png](attachment:data_matrix.png)\n", "\n", "In machine learning, the aim is to build a **model** from existing data i.e., existing features and responses. The model identifies some relationship between the features and the responses, so that with any future data you collect, you can make a good **prediction** (also called **estimate** or **hypothesis**) for what the observed response should be.\n", "\n", "A good example is house heating: if we have data on a number of houses (the **samples**), we can look at the amount of oil each house used in a year (the **responses**). We then build a model for the relationship between the usage and things such as size, insulation rating, occupancy, etc. (the **features**).\n", "\n", "Then, when we want to guess what the usage will be for a house with no existing heating oil data, we can **predict** the amount it should use according to our model.\n", "\n", "There are a large number of different statistical models or algorithms that can be used in machine learning to make predictions. When the prediction must be of a quantitative nature (i.e. where the responses are numeric values), we use a technique known as regression. When the responses are qualitative (i.e. when the responses are certain categories or classes), we use classification. \n", "\n", "For this assignment, we will be trying to obtain a numeric prediction, so we use regression. Specifically, we will use Ordinary Least Squares Linear Regression, the most basic form of regression." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Exploration\n", "\n", "In this exercise we will be trying to predict the heating oil usage of homes on a particular day based on a number of factors, including their insulation rating (1-10, higher is better), the temperature on that day, and the age and size of the home.\n", "\n", "We have two datasets for houses from two different locations. Dataset 1 includes the heating oil usage for each day, whereas dataset 2 does not.\n", "\n", "We will build a regression model from dataset 1 by splitting the data into _training_ and _testing_ data. The training data is used to build the model, and the testing data is used to see if it generalizes well. We will also explore a feature extraction method to see if we can improve the performance of our model, and verify this using *K-Fold Cross Validation*.\n", "\n", "Once we determine the model is performing well, we use it to predict housing heating oil usage for dataset 2. We then compare the two datasets and see why they differ.\n", "\n", "Before diving straight into the modelling, it is always a good idea to explore the existing data and see how the features relate to one another. This is a useful sanity check for later on." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set Up Libraries\n", "First, we must import the various python packages we need to use." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2018-10-02T12:05:34.928154Z", "start_time": "2018-10-02T12:05:25.940146Z" }, "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\leahy\\Miniconda3\\lib\\site-packages\\sklearn\\utils\\__init__.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n", " from collections import Sequence\n" ] } ], "source": [ "# display plots & graphs in browser:\n", "%matplotlib inline\n", "\n", "# Various libraries that are required\n", "import numpy as np\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "\n", "from sklearn.model_selection import train_test_split, KFold\n", "from sklearn.metrics import r2_score\n", "from sklearn.linear_model import LinearRegression\n", "from sklearn.preprocessing import PolynomialFeatures\n", "from sklearn.utils import shuffle\n", "\n", "# set the plot styles\n", "sns.set(style=\"ticks\", color_codes=True)\n", "\n", "def print_full(x):\n", " pd.set_option('display.max_rows', len(x))\n", " pd.set_option('display.max_seq_items', len(x))\n", " display(x)\n", " pd.reset_option('display.max_rows')\n", " pd.reset_option('display.max_seq_items')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Import Data\n", "Next, we must import the Housing data from a .CSV file as the variable `house_data`.\n", "In addition, we import the second set of data that we're going to be using to predict heating usage on as `house_data_2`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Import house data\n", "house_data = pd.read_csv(\"Household Dataset1.csv\")\n", "house_data_2 = pd.read_csv(\"Household Dataset2.csv\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Displaying the data\n", "Displaying data to get a good idea of what it looks like should always be the first step of any analysis.\n", "\n", "---\n", "\n", "The following shows the columns of `house_data`. Note the features, samples and targets.\n", "\n", "**Note:** The `print_full()` function just prints the data in a better visual format than the standard print function." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-03T10:12:37.351383Z", "start_time": "2017-10-03T10:12:37.329324Z" }, "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "print('Total number of samples in data:', )\n", "\n", "#Display first ten rows of the house_data.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As can be seen, we have 4 features, `insulation`, `temp`, `age` and `home_size`, and one target, `oil_usage`, with a total of 665 samples." ] }, { "attachments": { "correlation_strength.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiUAAACHCAYAAAG+IRmxAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABJBSURBVHhe7d0NkuQ2joDRPtCeYi6/x5oNjBceGA2ABEkxldXfi8BIJMGfVKlYdHXb8+t//+df/yb+Gf95KPgvHkogfCi/fv36/7v/0rqoTVU5Vb8R23dl/qxeRG28KYHfHkr0NH+y6PPypgR4KIHyoUSbUMdMv5U5dnNH/XlTAjyUAA8lwEMJ8FACUw/F/4TwV2u0s59i55H7rGzrZ/GmBHgogb8fCvHPuLMJfBEeSmD6oWS7eWeXl5yZvEzUd3b+zry8KYF/PJSdr+I3yj4vb0qAhxIoH8rsJpaRfjN9V8dfNZqPNyXAQwnwUAI8lAAPJTB8KDM/QWz7jZ8kOoeuzc/p2/V+Fm9KgIcS+M9Dkf8BMvJ+8KJg6NiLsvJDr7IzjvQ9tY5P6a7/6c977EWZ8e1fvD9Z+aL8hO9M5Dpf2/JFARQvCqbwomBK+qL4n1/RzzOtk6uGF9XNqMbSuWxZ2Xtv1KaxI+sf1XdyT9gZN31RAIsXBVN4UTCFFwVTtl8UOSA9dfj6aarD685z9P1Wx6lsvyjCfkh/FVW7vWb3/qr338au234Ovfftyrb5e71GYdt2HXlR8PP940UhiFHwohDD4McOSrwkGOIlwRAvCYbSl2T0z9on/lm849Z8K/NIn6fXd2L8aIyZcZd3klMPpRpH2+SqUYnaZ/pVRv2zOWfNjC3XzpjK94vGmBl7+SXB+42++LN4STCUviSn3kK8T/drG74kMz+n8L26X990JwEULwmGeEkwxEuCIV4SDF19SeyJ2p6wOydtr+q7M66wa7SycXfn25Wtd1f4kviJViau+uiHyXJm57NjRH1snW+P8j3JGY27Ihs38lRux9WdBN+JlwRDvCQY4iXBEC8JhnhJMMRLgiFeEgzxkmBo+SUZ/WZv57d/vt8Tv0X8hG/9HFsvif3Qtqz3WVnM1glbH7V/A13z6DPYtignq7N9bJxwZCfxC7OhonubE937sG3fxq5fr/Zz2HJ0na3Tq23ftfyS4M/BS4IhXhIM/f2SEEQVvCQEQRyJvzcTAOhiMwFwBJsJgCNam0n3D49W/pDp1B9MzfCfxZe9T6zNrsneR6q2HXbc0RpW6Jh2bHsfqdpm+TluzBnReZ8Yf3bsE3O3NhMrmryzoCzX1uv9KNe3Z/mrqnVkc/k+WV7F94nG9HXC3p9UjRuto8P3i8aJ5lidryOaV2Xzd9aVjav19mrrhc/xsnpRjSNXXzeyvJlEZicVdrEqKkd5Suuz9lOqeaq1aWh5h+1vxzw1/q6T68jG+NRnreat1jq7zmp8UbWP5qjabZufQ66+buToZvJp9gEAf4q3vPdbmwnfuMB3efJ7trWZ6A5I/FlxSjQ28dk4qbWZWE8sBsCznvy+Xd5MAMBiMwFwBJsJgCPYTAAc8dhmkv2iZ/TLn6xP1S9q17pRv0jVR2VjZ/Uqaxv1idqrPmq178zYKht/dt7buZFqXF+f5YqZXK2byRVZ3WzuLY9tJhH9oNkHHrWv6M51cm4vm1OszPvUWqt1ep1c8fSaR56evzP+6TVXa5A6G0+4upmI6oNo26kPe3OuGdWcq+tY6VetQ4zarU6u6ObN5otO7mnd9XbzTo/7hOubCYCfic0EwBFsJgCOYDMBcASbCYAj2EwAHMFmAuAINhMAR7CZADjiY5uJ/E09/7f7bJ3Qso2oPqrTsHydLdurv9ey8G32Pqqz91rGs+xz9l8DpeVRaK5e9V75PH/vw7apqF7vozq9t9c3+OhmorIHI2Xfll1Fdq+idn8V2b2Qsu8T5Y+ueE70jKOvQ1Tnr77Os/U+N2oT9l5I2ffx+Vm7zfu0j20mI/YhvemBAYi9djMB8F3YTAAcwWYC4IhwMyEIgtgJNhOCILaDjYQgiO3gdyUAlrGRANjGRgJgGxsJgG1sJAC2sZEA2MZGAmDb9EYi/2auRmX13+CdHf8UP0817601CfsM9N7WZUbtq+y4p+ewn0vvbV1m1D7Dj1GNeWK+zFNjy7gaM3bX0dpIZuwuaLf/LJlH57L3b7CylqfWb8d9Yo5PfVYZQ8ex9zf5OZ9aw+64M/2XNxIty1VDy/7q6yJRW9RX67QsfM4MO4a/t1cRtWlo2V5PsXOMnJ5b2XH13l41lM+Z1RmnO3bEzufv7VVEbRpatlcR1UU6fbL2LF9Ju+1ryzNmcrc3Ek/rbfvMQlRnXFW1ZXyuHyMaqxpf2qr2VdW4tq3K2xGN769WVDcrmkP5Nt/eFY0fXa1qTmmz7VVuRvtkfUftlZU+aqbv9kYyqpdrlmuNcqr2atyMHy+7CrnX0LKV1e/ojvXEGoQdz88RzVW1ZbrjrMzh+TGyq5B7DS1bUX2Wa0XjaES0Pmuv7PYd9ZveSN5s5eEAP8Ub3v8fsZGImV0T+Ene9M7/mI0EwOcsbyT89Ae+x9Pfr+2NRI9TbCTA93j6+3Z6I7ELIf6cOCEal/h8nDS9kYgnFwLgOU9/77Y2EvXUYgA84+nv2aWNBAAsNhIA29hIAGxjIwGwjY0EwDY2EgDb2EgAbGMjAbCNjQTAtqWNZPQ35KL20d+sy9qrPiLrk40nRm0jK32rOStRn9mxVvpm7Z2xorrM7ridXPHpXKnTsKpcr5Ob6eTOOL6RaNuJhY7GOjnXrGzOJ9aw8/l2+kZmx+nMeztXylGdvSpfFidyI0/lWj6/23/k1RuJuDXX7FhVXncdT8610zcy26ea16tyfd2bc6u+UZv3VK6SXA3V6T9jeiMZLULr/LUiOVVeNpavj8aI+kR5qhpL+PZqLCU5VZ62+xwt++uMlb6SM8qbGUd057XXym6u3GtYUW7mVK6vOzVuJZvzlOmNxKoWsfpBI6Oxbs6lqrwT67Bm1xTZ6RuZHacz7ydyfd2pXO8NuZXd/t7SRgIAFhsJgG1sJAC2sZEA2MZGAmAbGwmAbWwkALaxkQDYxkYCYBsbCYBtbCQAtrGRANjGRgJgGxsJgG1sJAC2sZEA2MZGAmAbGwmAbR/ZSE79Z95O/+fiKtlcWn9zLch1vh5Rzs2vYzZX5zO8xUc2EuEfkpTtA4zK9t6HtulV75XP8/c+bJve69Xf2zphy/6KZ0XPW+5tvc/RulF4tt7f63Wlzoats/dafouPbSTCPpzqKk7UKZ8T5Y5yRnnZFc+S56yh5egqRnW+PcoTp/Ky9tH1DV61kSgp+7Yo19ZF7crX+VzbHtWJrE/3imf55xw9f19XtQl7r6o+WV97L7J+3esbXN9I5MNHD8DW6/1sWfh7WxZZe3YvojYbWq/X6t7W4RnZM7b1o3tbVrbeynKyexG12dB6vUZt9v4trm8kAH4eNhIA29hIAGxjIwGwjY0EwDY2EgDb2EgAbPttIyEIglgNNhKCILaDjYQgCIIgiFcEhxKCIAiCIF4Rvx1KAAAAnubPHxIcSgAAwHX+/CHBoQQAAFznzx8SHEoAAMB1/vwhwaEEAABc588fEhxKAADAdf78IfHaQ4n+P4696f9tDL/j67OPZ/gz/QlfV97d7yVfO41P8ecPiSOHkuhD7XzQEw9pdgyfJ2Vbd2Itn+Y/k1W1zXjq+Tw1bke0htm6jjd81o5svd/yOWa/hruf5y3PQ9aRraVqm/HUZ3xiXD+m/+xPfZYTorXtrPfEZz0xhj9/SFw7lEhZw9KybbehojoR1ds6Wx+p2qNx7FXvhZZtndBy1Ca0Pmo7Rcf2c2T1Quo0PFvv27UtqterbdNyVpfV3+DnieaO1qJ5UW5Wr7KcN9G1+TVGa9bPkuVGbULro7ZdfsxonizH14uszZaznBt0Tj93Vi+kTsOz9b5d26J6vdo2LWd1Wu/LK6q+dnzNs1e9F1q2dULLUZvQ+qhtJBtPZWNr2bbpva0TUZ2I6m2dre/y5w+JY4eSapFVu71Xvs7209D6TNWWsWOrUVlUOattJ0VzdtYxmyv3PpS9z1T5Op6NG3QeO19Up6TOR8TW632W+zbR2oVff1Uetfk4Tce0Y0d1Qso+MrZN76v8G0Zr8uuryqM2H8reZ7J8HcvGrmicUVlUOattMyTfhxq1RXy97auh9ZmqbZY/f0g89psSq/vBfF3WvztuZDS/bx/li05/KWs8xY6t91GdqsqdNitq64xVjf0knTdaW7SmbJ2+PhpPZP3fJFqj1Pn6qtxpe4LOEc3r56/WU+Vm97dF66jWVpU7bVbUNjtWNW7HaA2+fZQvOv2lrNFV9Vlp8/WzeVbVNsufPyS2DyWyMI1KlDeqy+qtUf2I5mi+72PrshwRtfk6Xxa2ztafEo3t7327yOqFrfft2mbrozpl22yOvVc+55ZovmoN2TptfRQ+542q9WV1Pt/X+bKI6k6Lxs7mq9Zj26LwOTdF8/p73y6yemHrfbu22fqoTtk2m2PvhW9foX2zsWxdliOiNl/ny8LW2fqRmT5RTlQnbL1ti+rEqH6HP39IHPlNCdZEX9DdLzIA4F3Y62P+/CHBoQQAAFznzx8SHEoAAMB1/vwh8dihRH41ZQMAAJz1zT9r/flD4tihxD8YHwAA4Kzo563G2/nzh8TWoSR6CFkAAICzop+3UbyRP39IHP/jm+hhSAAAgLOin7cS38CfPyQe/4uu3/SAAAD4Jt/8M9afPyQeP5QAAAB4/vwhwaEEAABc588fEhxKAADAdf78IcGhBAAAXOfPHxIcSgAAwHX+/CHBoQQAAFznzx8SHEoAAMB1/vwhwaEEAABc588fEhxKAADAdf78IfHYoWTnvzLn+1XjdHIjb5yrWx+ZncvqjF/pzr0zb3cuITkrc3bm6uSO3Jy321/aZ+bojNvJFdI+yhGdcb8tV0nOKK877sy8ojuuWMnJ+uzWZ3niDbmZ0fgrY97izx8Sj/6mZPVh+H6jh2515+z21/buPKI7l9WdrzOXtsm1O0+kO/fOvJ25hLavzNedS63MZa3OK7pzr8y1klP1ecMavCp3ddxRXndcbT85rrRpjHTH1eto7M64XpXbHVfbZ/NUld/JzXTX8yb+/CHx9YcSIe02Onz+qL+2d+cR3bnU03OtriuzM1537tm5dtakdsZYmU+tzrsy58pcKzlVnzeswRrlrY4rqtzZcbvzd/OtKrcz7lO51iivO662PzGuja6Z8d/Knz8kfsShRK3M15lrdV1qtX93HtGZa3VdmZ3xunOvztWdR6zOJVbmU915b84lVnKqPm9Yg5gZU3THtarc1XFHeavjiiq3M+5TudYo76k1dHKt2Txv1G913Bv8+UPi8b9TsvOgo/5R2dd16RjR2F6WOyvrn423Oo/ozJXlrsrGi8pRXkc2RjXm6nyzc9k837YiGysq++jK+kZjZbmRE7lR3yw3kuVGZR+VLC8qZ7mRLLfqe3LcLC+T5Uf9s9xIlpv1nRlTdMbNciNZblT2dSfp+E/OscOfPyQe/U0JAABAxJ8/JDiUAACA6/z5Q4JDCQAAuM6fPyQ4lAAAgOv8+UOCQwkAALjOnz8kOJQAAIDr/PlDgkMJAAC4zp8/JDiUAACA6/z5Q4JDCQAAuM6fPyQ4lAAAgOv8+UOCQwkAALjOnz8kOJQAAIDr/PlDgkMJAAC4zp8/JDiUAACA6/z5Q4JDCQAAuM6fPyQ4lAAAgOv8+UOCQwkAALjOnz8kfuSh5NevX/8IFdXd8ql5n3bimf7UZwN02O8lHyfMjHVyvjfRz7Xz2X7qs/kkf/6Q+LG/KdGXx79IN14qP+dP1/m8f9qzATqi74+nvmeeGvetOp/3T3s2n+LPHxI//lAi7AuW1Quft9ImfFnM5uh1lCtsvc+ZLWudvfp6WxZVuWoTvixGOVlZ63w78K2id9nXVeWoTUVttixsXdauV9sW5Srblt2LrKx19lrVq6pctQlfFrbOt2dlrfPt+Is/f0j8EYcSoS+Frbd1NmybqspVmxrV2Tat9+Fl9cK3jcoqqtc6H6oqV21qlNMtA99q5l32OVnZ1oksz6pyonofkazN14/KalTvQ1Xlqk3ZOt/eLeMv/vwh8cccStToRdFy1l/4flXZXm2OiuqzOs/n+ftOWUX1WZ2y7T63KttrliO6ZeBbzbzLPseWs3pRle3V5ohOXcTn6n1UX5VVp96WbbvPrcr2Gt2Lbhl/8ecPiR97KNmlL1EUAABgjz9/SHAoKXAgAQDgGf78IcGhBAAAXOfPHxIcSgAAwHX+/CHBoQQAAFznzx8SHEoAAMB1/vwhwaEEAABc588fEr8dSgiCIAiCID4RHEoIgiAIgnhB/Ovf/wctegIiwuKC1wAAAABJRU5ErkJggg==" } }, "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Correlation Matrix\n", "Correlation analysis is a very quick but powerful technique that can be used to rapidly see how elements in a dataset interact and correlate with each other. This is particularly useful for the optimisation of building energy data as meaningful correlations between dataset attributes (e.g. HDD, footfall, daylight hours etc.) can be found quickly and effectively. In summary, correlation is a statistical measure of how strong the relationships are between the features and responses in a dataset.\n", "\n", "Correlation Coefficients between 0 and 1 indicate a **positive correlation**, whereas coefficients between 0 and -1 represent **negative correlation**. A positive correlation coeeficient between two variables means that as one variable rises, the other also rises. A negative correlation means that as one rises, the other falls. The closer a correlation is to 1 or -1, the stronger the correlation, whereas values close to zero indicate little to no statistical relationship between the two variables.\n", "\n", "![correlation_strength.png](attachment:correlation_strength.png)\n", "\n", "---\n", "\n", "We use the numpy (np) python package to get the correlation coefficients:\n", "\n", "```python\n", "c = np.coeff(X)\n", "```\n", "\n", "This returns a matrix, `c`, of correlation coefficients from an input matrix `X` whose rows are samples and whose columns are features. \n", "\n", "In the code below, `corr` is an array of correlation values for each column in the `house_data`.\n", "\n", "**NOTE:** `house_data` is imported with the rows and columns the wrong way around for the `np.corr` function. We need to TRANSPOSE the array by using the `.T` method" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:37:13.690146Z", "start_time": "2017-10-02T20:37:13.319269Z" }, "collapsed": true }, "outputs": [], "source": [ "# Create an array of correlation data. Note the .T for transposing\n", "corr = np.corrcoef()\n", "\n", "# Create the axis labels for use in the plot\n", "labels = \n", "\n", "# set the figure size\n", "plt.figure(figsize=(10, 8))\n", "\n", "# Plot a heatmap to easily visualise the relationships\n", "sns.heatmap(corr, xticklabels=labels, yticklabels=labels, annot=True,\n", " cmap='RdBu')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As can be seen, the darker the colour, the stronger the correlation (red for positive, blue for negative). From this, the following are some of the observations we can make from this graphic:\n", "* `insulation` & `oil_usage` are highly negatively correlated (-0.73) – a better insulated house uses proportionately less heating oil, and vice versa\n", "* `age` and `oil_usage` are highly positively correlated (0.87) – meaning older houses generally use more heating oil, and newer houses use less heating oil.\n", "* `home_size` and `oil_usage` are more weakly correlated correlated - the size of the home has some effect on heating oil usage, but not as much as age, outdoor temperature or insulation\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "Below, we can see various features plotted against `oil_usage`.\n", "\n", "As can be seen, the correlation signs and magnitudes match what is seen in the correlation matrix (i.e. strongly/weakly and positively/negatively correlated)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:37:22.824961Z", "start_time": "2017-10-02T20:37:22.367708Z" }, "collapsed": true, "scrolled": false }, "outputs": [], "source": [ "# plot age, insulation, home_size against oil_usage\n", "sns.pairplot(data=, x_vars=[],\n", " y_vars=[], kind='scatter', size=5)" ] }, { "attachments": { "over_underfitting.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsEAAADWCAMAAAAHI69MAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAARFQTFRFAAAAAAAAAAD/AAAAAAD/AAAAAABVAACqAAD/AAAAAAA/AAB/AAD/AAAAAAAzAABmAACZAAD/AAAAAABVAAB/AACqAAD/AAAAAAC2AADaAAD/AAAAAAAfAAA/AABfAACfAAC/AADfAAD/AAAAAAA4AABxAACqAADGAAD/AAAAAAAzAACZAADMAADlAAD/AAAAAACLAAD/AAAAAAAVAAB/AACUAAD/AAAAAABiAAD/AAAAAAA2AABIAAB/AAC2AADaAAD/AAAAAAARAAAzAABEAAB3AACIAAC7AADuAAD/AAAAAAAPAAAfAAAvAAA/AABPAABfAABvAAB/AACPAACfAACvAAC/AADPAADfAADvAAD/4u+jTQAAAEp0Uk5TAA8PHx8vLy8vPz8/P09PT09PX19fX19vb29vf39/f39/f3+Pj4+Pj4+fn5+fn5+vr6+/v7+/v8/Pz9/f39/f39/v7+/v7+/v7+/cW04AAAAU5klEQVR42u2dbYOjyHGAQQhHJCKOSCzFwznxCTteYkdE2BYZ4iTCTuxTzcvNzu7O3tT//yH5AEiAQALRDXRT/enm5m4kqh66q+tVUWiNaHkkAloiLwN0iZ9OVUnDsi8XXImfbrUiDUu+NIBI4n0qDEegwnETbAOALbGJBIb0KgxGbSmpEQBE8t5SAaS/qJpgjX0LBmlFoAEAyH7IehCMfQsGkNVYdAFA6otq8pYa4yU43oIBVtIqV/pN2AUAf+xbsKzuCDd+OKk3YRXGYCldU7Gk7ggtfTiZ9WvDCCylyqXDaUmoZS99NondEekpOtJNOMgQLJ8pZZweTt6bjiut+uqsFWSXdJe58PRs0gbmtBG8pFdsiMj1wHMBACLJDiI7+3rKGnX0j084Qp+wGgB4lqrYYCuq5csmAz13wEiaf2fKfIZet6CClRbvVbaiKIq2Cl3J3k8IbQ1As0MAkDJ3QM0YStKdodefXj+etukRK9M+5ULk6oqiACiKoruRlP4mJ3fMjDasYctoJFqeGf8DQHLeevLlfpgAAJ5lg22647QjJCb4uFKCZfRDRBCs1ESBquWD1MU2RLB8y3O0nAK1VaAqpiPoWeM7OhE8MoLPFWj4wprDOgCEDhE8coJjD6KQXhenhU+bCJbHDIyELVcIW1xFiWB5CBY2o19vk55EBMtDsAWCVj26bQLjRLA8BKuiRtCjNv5sIlgeguNMH0e4RzBbXUGJYIkIXomZS+q2qkMggiUiWBOyYENt50MhgiUiuJVXqrdltfNjE8EyEeyImO3utXMCEsEyEawLaEbEHhSTCCaCBTUjVi292ESwVAQ74rUXCFpGEolgqQg2hcvu0dqGYYhgqQgWL7vHaevDJoLlItgVzYwI27ZDIILlIlg0M8Jo7T0hguUiWDQzwm3twSaCJSPYFar0Xm3/whHBkhFsChXUsNobPUSwZAS3S7btevnty0qIYNkIFik3QmsZUa5F8GxOBAtFsEi5ETaDhOYrBKtbxMOMCBaI4PYe1u5WyKCo5DLB5v8hIu6lJXixvptIR7AjTKWGweK4uEiwBV8QEdGSlOAPiHiYyEawJkzBp8vCZL9EsA7wCRHx3ZeUYEREvJON4Hy21+RuvRzoV1eZTJS9QLAaATy8I+ILyEzwWjqCsxm3kwMifhjmV18xmWV4gWAPAODh4+uTuINGrhC8R0RcSkdw1kd1h4iIw7SUAiY9hqoJjs3sIBC5u/IVgmf7we5PrRTonRLU1oiIOEhvks5mClM1wfG4Od0Ws4tGPV+EMhPdFVGqwEysdjlcZ5LLxmdSSbCVzLGy+s0UWa7XE44Ei79KFZiJLH9A3A9yC1YZhb8rCQ6TN8To1bv4b+38XWMlOLu9TQYakLIYZTJXEWwePR0A0BsIP0ZExG9lJXi5x82EhwJ1AVzCAaNesVUE+8e3uMekf/UPiIj4Z0kJXiAi7rgoMBx8K2GD1TTdCoL1k7M56G9urwU/ICI+m3ISvEFERC4KZONqZbAmm8N+yfEeV0lw5u/7/REcwNMPiK+3u0JGSzCbcBeDtUNEnFd9wRU3guN7on2CuZdUJ6313OthEzxHBvGyS1vQALKEq8Ke7A6JcgFk74l2bwSvjiNXJb3Jzbe7NR8FJjdxfbAEszPUywWQLf6we7sU+K1n04/Vmzacu9ymPCTI8AUrFUCu+MPoK6QRm0pRmzqUERNsD+MuN1nvtvOKzcnnJ4BcN/veCDYBACKnjREzYoI1GHDjCJZfrlQAQTYXQu9rxpMDAOC2MmJGTPBg7nKV3y3iJ4BCkn9fQbkAAMBqdQSMmWBmIQMO9mHE0DtQJoDCSJyeCI7NYK3VETBmggccl1uxLKcuE0BhLFnYz7tsJOi2eYFGTfBqsGX3TN+tEgFM3hA/ZTwdPQXl7CRNO2rhdxk1wUzPapbLYmrflAhgU8im8dv3Vbll+UnYsc0LVEbwbD4bB8FM70vMNetzFMChkDLl9PMqRwm5/pnjZbpeT28meIPiVxbVJHigDjWD7dcqEcAeEXFbOM47J1hL7d+zj59XJIvUIng23LoxdgRP5vPJcbMbnEPNZVszUSKAJSLionAj6LyxvZlK/6xQb1c/rfac4LjyccC94Gbbw3bWToGzQ9IqzByiQ00DtrXDZa/wYruZF3d9v/u9JQHXLH58K4LjPXg6WIAnh4Z1VSUK3B4P0XCAUzVcxuHuGt1XjV7OoqP5e/YC/Tci4qaNHTzgJidzLJyANyjw9Ipbw3OoqRHAx7fv1l0SrPYS0kgvcmdRbe3hDfFtcivBA/dFxAQvWynwtAfHDrVBRTVsgBdkeZuu0wG7D4K104cWPt4CeHyseSQI6A/et7ciZvtjy1x7aKON1AjgDRHx0CXBfQTlzJPpUlCC06AFi4AET7e7tje5oy8i3QgGFNVYQR8E9xGUy6SkFUpNgwbexFHH5Ljcm9quEAC+79yK6KNSzju5XAovUJPLCRHM2nfVcsVXy3/e7Tq9ySl9tE7LWC5eTgdGk1ApERxLbzAd3UP2ntk6BPfROq2y1NRu4uIkglnHcFlswUbnBPfQOi270+aPAK+JSUMEZ7svSbkF15sn1707Lbvt54NyUZO3mAge1CbMYwuuR3D37rSs5ZALyumN3iYieFCbcMjjm9Qi2O/cGZFNqdSy0K4ahbiJ4OPON4BNmM8XqUWw3XmCSM5WyBLsNfKLEMHctr6hbMH1CDYA4PE5F+qczrn278/ZCtk6o2YNUIjgAW3CnL5GvcngEXxGxJdjFyj4zHcIUN77kQlp6M3i/ETwgDZhTt+iHsHuMyIiPqQAxz/+bSeuiFPFnNJ4cHtdguf5uqV57TImMQgexCZscUq3r0ew8YqIiM8pwfGP/6vyU8tZEMM+vccr1gQv85VHS5EKkeo7k/rdhONET78nASjKrxER8TEl+CX+kdtb7X//+nTaM05lTnrDjO2aBBcqA4uFguITbAE8fHz9VebqMrnreCi6zaviqSbByj0ifjrZwV8R8RO/SPP/IOKrkbOKfUVp3gusJsGYr1tCFvMtBkWw4sPX/DPdd/yI3Lbg2gQri3WuPPIfX5/5xemWiIj4V+mPR4dw40Srmt9wm2/TvOU8bnm52y46VqDxnK9wnXdd8OpwKzq1b41URBzjdMWC4tR2aJzsWpPgyQ5xO6n6kcvTLTpW4O/zIr3rmOB47+ESUriZYJ9j3umiMM46iJ3AcfF4k6qvQXrTDkzN7JoK/BtExPdfpD9OOZdsTze73KQ8j1/J6c0E80watj8j4q9PP8cp9mbUuN5guATvulbgv7/j+8uJoeUBD/wc+pM9It5njBjgV3F6M8Emx6RhG56eHzPfawXPX/HtqXnN1yAJ3iAi3nWtQI7neNUhOs8f2JxqnW4mmGfScLFVmv74jojvzftWDJLgyQYPhXvifM5fgU6HDXyWeYItnvWmdosZFdycEWelpf+VBlRCVWCCjcKyk/VPe8T7b2zb8/MrqhNAr63A2J/VTeeaaa7zkBoHVNRBEsynHcxZy+CfJgQHqtIfwWoBQNMurAKAp2FiV9YXRMTP5b8z2CnQ7jC2PN/j/Sz3wY+/nA6NYI41+Gfb++SAiF9Du/kfOjMHTwRaBQCdAoABdLQQEfGNO8FxbLn7ynstzUNYj4bgkjZX083uw49ueRXKN6JhrTdExC/8CY69kU7XBPsA8MSv6e3tBPPrIsGu1aAYBD+9I74/8Sc4GZLa8aha85TMOB8WwTZfgn35CA4KZoqb2i8/+/bn39i2bRUMbY21AmOPWrd9E+Jr3J+GuAfzC2lYHAletSCweFMr3uTMAoGdWJyNFGh3nygcO/FWm9odc7sjmF9Ig93ufk6wfgLQKxJY9HYpIqxGClQ7v8zp6bY/X8+VgRFsCEmwdKuZAk3ouGg34G56304wv3nLHrOkISK4/DJndPftuLs/bieYX1COnZ+OCD67zEU842PlV0e+n9aWYI0IForgLnbFbnf8FgQHnxDf/prDd4qYmU5EcJVlanTz3ToY49GC4N/xKidjZ560+TOz9UJKgnXoyo6IP4m356MFwTtERJSV4CUi3k8kJLibnfG025vKYAneICK+s3/DGGYe307wpOlULHEIzpC1vOPY2cXpxnPXguDZARFfLT4E+z0THBfzrqUkOD3dJ/c8X1KjI2ulBcHKv76+fs/hHfvm89vbs9czwXE1m5x7cGJH+HdMp2IVVpxQ38GNsQ3BJp9xe/eIiD8bgh3MJZA/nfevwNjL9WdeF5nTRzjKoAlu2Aq1rnHC8PxucyGc8pleu0Xcz/pWYBzX+MhxD463+S5qmloR7PG41P7dQAjms+4wX4fekwKtNLX+wMVlOP/72NTWh06wxSM1wv+KiIeppAQzd0HeqEAXAAD+k0sf89khKfvrJI2zFcHqzcb6Yj2rvMI+fH778g+KpAR/YH1y36hANeBnp+4xbpfuKoMnODbXb5DCBTcO08k7wyN4emDs4rhVgXrELdsdERFfGxeW90Lwqhq3S7eVf0FExB+Vb8Es+7H1Q/Ddblvtb5jcsU31vlmBsSnMI3UXERE/Rh3V47UjWKsSwmx/YXq5FreA/0PZ79g2x+iF4A+dtoW8XYFxzCxinl1ovSLi1wdTEYHgOEB5/heM7zDfuy9/i3hBRHx/sCs2dXZHGyeCJ8v1hSPm0GkL+BYKjF22Nx32083uw6TSPHl+/fhgK2IQbJd6/fQoNoUqTPl0MtK5FyMO5LDrxsaH4CvR2G5bwLdQYFw1d4vTdnKo9Akm9rWrCEJw6WALNYTEmC+1Z00AeHp+LNtrXcaRSD4EX4nGbjqNR7dR4CXapuvl5LIESi2l9K1QRSG4dLiQDfAJEd8fywN27qmBQuk1jmH1KB+C15d9uiW9KYeqwDgvoAzh5UWnfCXBiZOuy2rotgSXDHjTAABeXl8fKu4JIQCAV3IHTN5fbegEL3lmxHSrQKsK4T3ihbv4FLO9Kc8B1hVxCC4xI2I4Da/CyErGcobnrmSHeZtZTje5DeJhLgXBiczPEb5izC/2uJtVAdxtX6u2BJ+bEWnH+aqoTzKd3j6LSCcJpcrwCVZm88FUb7RVoFuO8P6GRvMpwJZQAjg3I44d5+MEqDNTOJlOrxV/mdgQhggED2i1VmA5wrMD4m4iAMDtBVA0I4yTKWCWuszTlEyvUIPicgjUE8G378JNk0u1fgBmIICCGZEd+uGU+Ba0NC3eyufHmzyKUojgJgi3kn3imet+/nh7ATg569XI3caC87uZdbQ6ouwDa1yKUojgJgi3ceKafQHMQAB67vrp5dyBCZd6EXjnKLcgZ0OxDkUSwY0QDm92IqRNbS0hBRBmrCitAOK5bRAcH9TIWNAun6IUIrgZwtFtBKrp/26KKYBVxi92NvnYLdwS1Ay2J5fwilOaFBFc/8/Ey7nBktCTO1ykCyoA7XR+aGe2QOIjM3N2cpiRWqQcI0Ps32AiuPay0kh/YwxXiQkcaMIKwDtaACXD5/X89mpnfGgp+vF/wmMmBxHc3JnQ8O9p6dg8TxVXAGbqRjDKJGDnXGp+1vfmATx8/Mu3f+SWj0cEN7Fm0wF6QQOPkH0b90MTQJgwWj4A2s8+Yc43YcSz1L5yy8cjghstJ00arOsaNsLkf4gMsQUQX8QMs/xlzLp6jXxfyjCelPc9r4TSvgleH3AzEYdgxUh31MiuoQ/jOHe3LwuCmQCSodNRRWpoxqVm54N0Vjwp7yOvjOieCV4jtxlUfAhWVA+ODF+5mJlHfqNVn0JmIwAb4OELvr9W+LTdYx9Ov7BL//YdEfGRV0p/zwTvOkgkthlboGZ4LEBwq51D2ir9z57ecL8Qn2A1jAe0v5QHJRKXmp2YwRmbSf2PH/D9hdshxI7g2Xx6I8F7sQhW1MzY09AxS/HNDE3/ARFxKjzBipkMaK9waif+sthSziUFqyvP5ncLYEbw5qbSt0UHPYht9l4Azc1NMrWtk4I0w/bCzC/DX/beZ5mVAJx3RMQv1sW7XuR3PI+PFcELvG2nWWx3S0UUgifzeTnD6UjeoPjvQivpNHonAcHKbxARf1L5axcA4On15aHT7A9WBK+x0y4mvShwdkDcH19SzYmujpm2UkvpMJVBAMryUq8lRQ0AXhDx6wOo4hG8xL6tPf4K3BYdJ5Z3Ad/QSVwVk/VuM5NCAFeNqwjeL7VBGbYdvL1+VM62u/VEYAWW1Haqllu6E/u2LuErfHUljXzedBEJVuZ3Vzaa6aHLTj0cFHhf7rzWLDtjAEe+uzKUQa3uCFb07xARf6UISXBNU3kqrgJnh0sGrWYYhq6sd7vBjYnskOC4/lWRmuCZwAqcLBZXrKAPiIiLEROsTOYda7hDguesB2QMUIHdduUcIsGdry6jyot73E7kVmC3XTmJYIWyKxmvzRCnRBLBRHB9K3Bz2K8VIpgIJgWSAIhgIpgIJgWSAIhgUiAJgAgmBZIAiGAimAgmBZIAiGBSIAmACCYFkgCIYCKYCCYFkgCIYFIgCYAIJoKJYFIgCYAIJgWSAIhgUiAJgAgmgolgUiAJgAgmBZIAiGBSIAmACCaCiWBSIAmACCYFkgCIYFIgCYAIJoKJYFIgCYAIJgWSAIhgUiAJgAgmgolgUiAJgAgmBZIA6izLIIKJYJGXVv0IEhCsEcHyCyAAX5OW4EAngqUXgAUQrWQl+JqCZDhER0+wCgAQGJJaEVB5wsiiQNNVR06wEo+w9jQpb3IBQGTLrcAo0EdOsJVM/XU1CQm2AABCU2YFOtDiIeS4CBwnsLu6dASryTx5Q14F6lVWYD2CXUP85R+Hr4NvSUZwYiNVMSzFFhSWnqB1CZZuRZIRbJ7eTlNSglex4uxbbnSWL8MKMwB7ZpZgGZ4u83DhSpWRYC3dfGxVGec6Ehw5Wt4Olm4FqowXmeCoQFcbI8B61StsyLDMjIF0rl8bQokP0bEsR+ojaAWXzlgZLzKr0REcAQC4sppQyQkbWuUmpBTnjJcx9a3x2cImALSJ6ohwy5H8jnO0InxDGeFypT53HAAAR+59SR81v4oaybsBx9uT1M8X70Ej5ldRLFfmDcqAESQARxdTP6RfcjsQ3VCXXoMmAIw2liG/kT8CzXry20m0pL7IgENCoCXyRcYiGeTX/wMciGAh9KpqDAAAAABJRU5ErkJggg==" } }, "cell_type": "markdown", "metadata": {}, "source": [ "# Building a Machine Learning Model\n", "\n", "## Linear Regression\n", "Linear Regression is a simple but powerful technique that is used for numerical prediction. It is a statistical measure that attempts to identify the strength of correlations between one dependent variable (the response) and a series of other changing instances known as independent variables (the features).\n", "\n", "Linear regression models are simple models and often provide an adequate and interpretable description of how the inputs affect the outputs. For prediction purposes they can sometimes outperform more elaborate nonlinear models, especially in situations with small numbers of training samples, low signal-to-noise ratio or sparse data.\n", "\n", "The equation of a linear regression model is simply the sum of the features for a particular sample, except that weights are applied to each feature before summing them together:\n", "\n", "
$\\hat{y} = w_0 + w_1x_1 + w_2x_2 + ... + w_mx_m$
\n", "\n", "Where:\n", "\n", "* $\\hat{y}$ is the prediction/estimate of the linear model for a particular sample (as opposed to $y$, the actual value)\n", "* $x_1,x_2, …, x_m$ are the $m$ features of a single sample\n", "* $w_0, w_1, …, w_m$ are the $m$ associated weights applied to each feature\n", "* The $w_0$ weight is also known as the _intercept_. This is a constant that is added to the equation, and is the value for $\\hat{y}$ if all the features $x$ are equal to zero.\n", "* There is no $x_0$ variable. Sometimes, to make mathematical operations easier, $x_0$ is included and set as 1, so that $w_0x_0 = w_0$. This is known as a \"dummy\" variable.\n", "\n", "How it works (at a high level):\n", "* select some weights, $W = \\{w_0, w_1, ..., w_m\\}$.\n", "* Get estimates, $\\hat{y}$ for every sample in the dataset based on these weights\n", "* Compare the sum of the errors between the estimates $\\hat{y}$ and the actual responses $y$\n", "* Optimisation algorithms search over lots of different **$W$** until this error is minimised\n", "\n", "In this exercise, we use the popular `sklearn` package in python to perform linear regression.\n", "\n", "\n", "\n", "## Evaluating Model Performance - Coefficient of Determination ($r^2$ value)\n", "We evaluate the performance of our model with a particular _scoring metric_, which measures how well our features **$X$** predict our estimate, $\\hat{y}$, in the model. For linear regression, a commonly used metric is the $r^2$ value, which is essentially a similarity score with values between 0 and 1. Typically, \"good\" $r^2$ values are above 0.75.\n", "\n", "## Over- and Underfitting\n", "When training on the data, a low $r^2$ value indicates that the model is not entirely accurate. This is known as **underfitting**. However, a high $r^2$ value does not guarantee that the model will perform well on unseen data as it may **overfit** the data. Examples of these can be seen below:\n", "\n", "![over_underfitting.png](attachment:over_underfitting.png)\n", "\n", "\n", "Each point on these graphs represents a training sample, with a single feature, $x$ (on the x axis), and a target value, $y$, on the y axis. We want to generate the equation of a line which will fit these samples fairly well, shown in black. These are the estimated values of $ŷ$ which will be generated for any unseen values of $x$.\n", "\n", "In the first example, we see that the model is not capturing the true relationship between the feature and target. This leads to a low $r^2$ score.\n", "\n", "The third example shows overfitting. In this case, the model would generate a perfect $r^2$ score. However, it is obvious that this is not an accurate reflection of the general relationship between the features and targets.\n", "\n", "The middle example shows how we would expect to fit the data. This has a lower $r^2$ score on the training data, but is still a better model overall.\n", "\n", "So, how do we evaluate a model to make sure that it generalises well to unseen data (i.e. the future values we will be predicting)? For this, we split our existing data into a _training_ set and a _testing_ set.\n", "\n", "## Splitting Data into Training and Testing Sets\n", "\n", "\n", "To build a successful model that generalises well, we need to split up the initial data we use to build and evaluate the model into **training** and **testing** data.\n", "\n", "The training data is used to build the model. The machine learning algorithm will keep iterating over the samples and making target predictions, and compare these to the real value of the targets. On each iteration it will slightly change various parameters specific to the algorithm in use, and keep iterating until the training predictions it makes closely match the actual training responses.\n", "\n", "In order to verify that the model performs well on new data that the model has never seen before, we use the testing data to verify. The testing data is fed into the model and a score is given for how similar the test predictions are to the actual test responses.\n", "\n", "Typically, a ratio of about 80/20 is used for the train/test split.\n", "\n", "---\n", "The code below gets our features, `X`, and the associated targets, `y`.\n", "\n", "The first 80% of the samples are then used for training, and the rest for testing." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:37:32.054563Z", "start_time": "2017-10-02T20:37:32.039518Z" }, "collapsed": true }, "outputs": [], "source": [ "# Create a matrix of the features (insulation, temp, age, home_size)\n", "X = \n", "\n", "# Create a vector for the dependent variable/target values (oil_usage)\n", "y = \n", "\n", "# Create an array of indices of the training and testing data. The first ~80%\n", "# of the data is used for training, with the rest for testing\n", "train_indices = np.arange(0, len(X) * 0.8, dtype='int')\n", "test_indices = np.arange(len(X) * 0.8, len(X), dtype='int')\n", "\n", "# shuffle the data\n", "# X, y = shuffle(X, y)\n", "\n", "# get the training samples\n", "X_train = X.iloc[train_indices]\n", "y_train = y.iloc[train_indices]\n", "\n", "# and the test samples\n", "X_test = X.iloc[test_indices]\n", "y_test = y.iloc[test_indices]\n", "\n", "print('train and test sets created')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Building the Model\n", "\n", "### Training\n", "\n", "The linear regression model to predict the heating oil consumption of a dwelling is as follows:\n", "\n", "
$Estimated Heating Oil Use = w_0+(w_1*InsulationRating)+ (w_2*Temperature)+ (w_3*AgeOfHome)+(w_4*SizeOfHome)$
\n", "\n", "We use statistical packages which use specialised optimisation algorithms to search for and find the best weights. The weights are found by feeding the training data into the model. This training data consists of the measured features, $X_{train}$ and responses, $y_{train}$. The associated weights, $w$, are found by minimising the difference between the estimations, $\\hat{y}_{train}$, and the actual targets, $y_{train}$.\n", "\n", "---\n", "The code below trains a new linear regression model, `oil_use_model`, on the training data, `X_train` and `y_train`. This results in finding the weights, $w$.\n", "\n", "The intercept, $w_0$, and weights, $w_1, w_2, w_3, w_4$, are printed below. Note that weights are sometimes called _regrsssion coefficients_, which is why they are stored here here in the `.coef_` attribute." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-04T14:16:14.390164Z", "start_time": "2017-10-04T14:16:14.199665Z" }, "collapsed": true }, "outputs": [], "source": [ "# create an instance of the sklearn linear regression model\n", "oil_use_model = LinearRegression(fit_intercept=True)\n", "\n", "# fit the model to the training data - this is where optimisation is done\n", "oil_use_model.fit()\n", "\n", "intercept = oil_use_model.intercept_\n", "weights = oil_use_model.coef_\n", "\n", "# the 'format' function here is just to format to 2 decimal places\n", "print('intercept (w_0):', intercept)\n", "print('weights (w_1, w_2, w_3, w_4):', weights)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this instance, our linear regression model, `oil_use_model`, consists of the following equation:\n", "\n", "
$Estimated Heating Oil Use = 131.46 - (3.76 * InsulationRating) - (1.19*Temperature) + (2.15*AgeOfHome) + (3.09*SizeOfHome)$
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Testing\n", "Once the model is built, we use the weights found previously along with the testing data to verify its real-world performance. As we know, the testing data has a number of samples, again consisting of a number of features, $X_{test}$, and responses, $y_{test}$.\n", "\n", "We substitute the $X_{test}$ features (home size, insulation, etc.) into our regression model, and multiply by the weights found previously to get an estimated $\\hat{y}_{test}$ for each sample.\n", "\n", "We can then compare our new $\\hat{y}_{test}$ estimates to our actual measured $y_{test}$ responses to calculate a test set $r^2$ score. This $r^2$ score is the performance we’d expect our model to have with new, unseen, data.\n", "\n", "---\n", "The code below gets predictions, `y_hat`, using the weights stored in our trained model, `oil_use_model`.\n", "\n", "It then calculates the $r^2$ score using a built in function from `sklearn`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:37:42.295852Z", "start_time": "2017-10-02T20:37:42.287858Z" }, "collapsed": true }, "outputs": [], "source": [ "# find predictions using the test data\n", "y_hat = oil_use_model.predict()\n", "\n", "# get the r2 score of actual vs. predicted y\n", "\n", "score = r2_score()\n", "\n", "print(score)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualising the Predictions\n", "\n", "The resulting score is ~.82, showing that our model fairly accurately models the data.\n", "\n", "By plotting the predicted $\\hat{y}_{test}$ against the actual $y_{test}$, we can visualise how accurate our model is.\n", "\n", "---\n", "Below we plot `y_hat` against `y_test`. Note that we have re-organised `y_test` here to be in ascending order for easier visualisation. Otherwise the plot looks quite messy (you can play around with this yourself)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:37:45.291007Z", "start_time": "2017-10-02T20:37:45.037333Z" }, "collapsed": true }, "outputs": [], "source": [ "# Create vector of numbers from 1 to length of y_plot, to serve as the x-axis:\n", "nums = np.arange(len(y_hat))\n", "\n", "# sort the values in ascending order\n", "y_test_sorted = np.sort(y_test)\n", "y_hat_sorted = y_hat[np.argsort(y_test)]\n", "\n", "# difference between y_hat and y_test\n", "delta = abs(y_hat_sorted - y_test_sorted)\n", "\n", "# create the plot\n", "fig, ax = plt.subplots(figsize = (10,8))\n", "\n", "# Plot the delta as a line graph\n", "ax.plot(nums, delta, label = 'delta (|y_test - y_hat|)')\n", "\n", "# Plot y_hat\n", "ax.scatter(nums, y_hat, color='r', label='Predicted/Modelled Values')\n", "\n", "# plot y_test\n", "ax.scatter(nums, y_test, label='Actual Test Set Values')\n", "\n", "# Set up legend & title\n", "fig.suptitle('Plot of y_test vs. y_hat', fontsize='xx-large')\n", "ax.legend(fontsize='x-large', frameon=True, shadow=True)\n", "ax.set_xlabel('sample no.')\n", "ax.set_ylabel('predicted/actual heating oil usage (kWh)')" ] }, { "attachments": { "over_underfitting.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsEAAADWCAMAAAAHI69MAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAARFQTFRFAAAAAAAAAAD/AAAAAAD/AAAAAABVAACqAAD/AAAAAAA/AAB/AAD/AAAAAAAzAABmAACZAAD/AAAAAABVAAB/AACqAAD/AAAAAAC2AADaAAD/AAAAAAAfAAA/AABfAACfAAC/AADfAAD/AAAAAAA4AABxAACqAADGAAD/AAAAAAAzAACZAADMAADlAAD/AAAAAACLAAD/AAAAAAAVAAB/AACUAAD/AAAAAABiAAD/AAAAAAA2AABIAAB/AAC2AADaAAD/AAAAAAARAAAzAABEAAB3AACIAAC7AADuAAD/AAAAAAAPAAAfAAAvAAA/AABPAABfAABvAAB/AACPAACfAACvAAC/AADPAADfAADvAAD/4u+jTQAAAEp0Uk5TAA8PHx8vLy8vPz8/P09PT09PX19fX19vb29vf39/f39/f3+Pj4+Pj4+fn5+fn5+vr6+/v7+/v8/Pz9/f39/f39/v7+/v7+/v7+/cW04AAAAU5klEQVR42u2dbYOjyHGAQQhHJCKOSCzFwznxCTteYkdE2BYZ4iTCTuxTzcvNzu7O3tT//yH5AEiAQALRDXRT/enm5m4kqh66q+tVUWiNaHkkAloiLwN0iZ9OVUnDsi8XXImfbrUiDUu+NIBI4n0qDEegwnETbAOALbGJBIb0KgxGbSmpEQBE8t5SAaS/qJpgjX0LBmlFoAEAyH7IehCMfQsGkNVYdAFA6otq8pYa4yU43oIBVtIqV/pN2AUAf+xbsKzuCDd+OKk3YRXGYCldU7Gk7ggtfTiZ9WvDCCylyqXDaUmoZS99NondEekpOtJNOMgQLJ8pZZweTt6bjiut+uqsFWSXdJe58PRs0gbmtBG8pFdsiMj1wHMBACLJDiI7+3rKGnX0j084Qp+wGgB4lqrYYCuq5csmAz13wEiaf2fKfIZet6CClRbvVbaiKIq2Cl3J3k8IbQ1As0MAkDJ3QM0YStKdodefXj+etukRK9M+5ULk6oqiACiKoruRlP4mJ3fMjDasYctoJFqeGf8DQHLeevLlfpgAAJ5lg22647QjJCb4uFKCZfRDRBCs1ESBquWD1MU2RLB8y3O0nAK1VaAqpiPoWeM7OhE8MoLPFWj4wprDOgCEDhE8coJjD6KQXhenhU+bCJbHDIyELVcIW1xFiWB5CBY2o19vk55EBMtDsAWCVj26bQLjRLA8BKuiRtCjNv5sIlgeguNMH0e4RzBbXUGJYIkIXomZS+q2qkMggiUiWBOyYENt50MhgiUiuJVXqrdltfNjE8EyEeyImO3utXMCEsEyEawLaEbEHhSTCCaCBTUjVi292ESwVAQ74rUXCFpGEolgqQg2hcvu0dqGYYhgqQgWL7vHaevDJoLlItgVzYwI27ZDIILlIlg0M8Jo7T0hguUiWDQzwm3twSaCJSPYFar0Xm3/whHBkhFsChXUsNobPUSwZAS3S7btevnty0qIYNkIFik3QmsZUa5F8GxOBAtFsEi5ETaDhOYrBKtbxMOMCBaI4PYe1u5WyKCo5DLB5v8hIu6lJXixvptIR7AjTKWGweK4uEiwBV8QEdGSlOAPiHiYyEawJkzBp8vCZL9EsA7wCRHx3ZeUYEREvJON4Hy21+RuvRzoV1eZTJS9QLAaATy8I+ILyEzwWjqCsxm3kwMifhjmV18xmWV4gWAPAODh4+uTuINGrhC8R0RcSkdw1kd1h4iIw7SUAiY9hqoJjs3sIBC5u/IVgmf7we5PrRTonRLU1oiIOEhvks5mClM1wfG4Od0Ws4tGPV+EMhPdFVGqwEysdjlcZ5LLxmdSSbCVzLGy+s0UWa7XE44Ei79KFZiJLH9A3A9yC1YZhb8rCQ6TN8To1bv4b+38XWMlOLu9TQYakLIYZTJXEWwePR0A0BsIP0ZExG9lJXi5x82EhwJ1AVzCAaNesVUE+8e3uMekf/UPiIj4Z0kJXiAi7rgoMBx8K2GD1TTdCoL1k7M56G9urwU/ICI+m3ISvEFERC4KZONqZbAmm8N+yfEeV0lw5u/7/REcwNMPiK+3u0JGSzCbcBeDtUNEnFd9wRU3guN7on2CuZdUJ6313OthEzxHBvGyS1vQALKEq8Ke7A6JcgFk74l2bwSvjiNXJb3Jzbe7NR8FJjdxfbAEszPUywWQLf6we7sU+K1n04/Vmzacu9ymPCTI8AUrFUCu+MPoK6QRm0pRmzqUERNsD+MuN1nvtvOKzcnnJ4BcN/veCDYBACKnjREzYoI1GHDjCJZfrlQAQTYXQu9rxpMDAOC2MmJGTPBg7nKV3y3iJ4BCkn9fQbkAAMBqdQSMmWBmIQMO9mHE0DtQJoDCSJyeCI7NYK3VETBmggccl1uxLKcuE0BhLFnYz7tsJOi2eYFGTfBqsGX3TN+tEgFM3hA/ZTwdPQXl7CRNO2rhdxk1wUzPapbLYmrflAhgU8im8dv3Vbll+UnYsc0LVEbwbD4bB8FM70vMNetzFMChkDLl9PMqRwm5/pnjZbpeT28meIPiVxbVJHigDjWD7dcqEcAeEXFbOM47J1hL7d+zj59XJIvUIng23LoxdgRP5vPJcbMbnEPNZVszUSKAJSLionAj6LyxvZlK/6xQb1c/rfac4LjyccC94Gbbw3bWToGzQ9IqzByiQ00DtrXDZa/wYruZF3d9v/u9JQHXLH58K4LjPXg6WIAnh4Z1VSUK3B4P0XCAUzVcxuHuGt1XjV7OoqP5e/YC/Tci4qaNHTzgJidzLJyANyjw9Ipbw3OoqRHAx7fv1l0SrPYS0kgvcmdRbe3hDfFtcivBA/dFxAQvWynwtAfHDrVBRTVsgBdkeZuu0wG7D4K104cWPt4CeHyseSQI6A/et7ciZvtjy1x7aKON1AjgDRHx0CXBfQTlzJPpUlCC06AFi4AET7e7tje5oy8i3QgGFNVYQR8E9xGUy6SkFUpNgwbexFHH5Ljcm9quEAC+79yK6KNSzju5XAovUJPLCRHM2nfVcsVXy3/e7Tq9ySl9tE7LWC5eTgdGk1ApERxLbzAd3UP2ntk6BPfROq2y1NRu4uIkglnHcFlswUbnBPfQOi270+aPAK+JSUMEZ7svSbkF15sn1707Lbvt54NyUZO3mAge1CbMYwuuR3D37rSs5ZALyumN3iYieFCbcMjjm9Qi2O/cGZFNqdSy0K4ahbiJ4OPON4BNmM8XqUWw3XmCSM5WyBLsNfKLEMHctr6hbMH1CDYA4PE5F+qczrn278/ZCtk6o2YNUIjgAW3CnL5GvcngEXxGxJdjFyj4zHcIUN77kQlp6M3i/ETwgDZhTt+iHsHuMyIiPqQAxz/+bSeuiFPFnNJ4cHtdguf5uqV57TImMQgexCZscUq3r0ew8YqIiM8pwfGP/6vyU8tZEMM+vccr1gQv85VHS5EKkeo7k/rdhONET78nASjKrxER8TEl+CX+kdtb7X//+nTaM05lTnrDjO2aBBcqA4uFguITbAE8fHz9VebqMrnreCi6zaviqSbByj0ifjrZwV8R8RO/SPP/IOKrkbOKfUVp3gusJsGYr1tCFvMtBkWw4sPX/DPdd/yI3Lbg2gQri3WuPPIfX5/5xemWiIj4V+mPR4dw40Srmt9wm2/TvOU8bnm52y46VqDxnK9wnXdd8OpwKzq1b41URBzjdMWC4tR2aJzsWpPgyQ5xO6n6kcvTLTpW4O/zIr3rmOB47+ESUriZYJ9j3umiMM46iJ3AcfF4k6qvQXrTDkzN7JoK/BtExPdfpD9OOZdsTze73KQ8j1/J6c0E80watj8j4q9PP8cp9mbUuN5guATvulbgv7/j+8uJoeUBD/wc+pM9It5njBjgV3F6M8Emx6RhG56eHzPfawXPX/HtqXnN1yAJ3iAi3nWtQI7neNUhOs8f2JxqnW4mmGfScLFVmv74jojvzftWDJLgyQYPhXvifM5fgU6HDXyWeYItnvWmdosZFdycEWelpf+VBlRCVWCCjcKyk/VPe8T7b2zb8/MrqhNAr63A2J/VTeeaaa7zkBoHVNRBEsynHcxZy+CfJgQHqtIfwWoBQNMurAKAp2FiV9YXRMTP5b8z2CnQ7jC2PN/j/Sz3wY+/nA6NYI41+Gfb++SAiF9Du/kfOjMHTwRaBQCdAoABdLQQEfGNO8FxbLn7ynstzUNYj4bgkjZX083uw49ueRXKN6JhrTdExC/8CY69kU7XBPsA8MSv6e3tBPPrIsGu1aAYBD+9I74/8Sc4GZLa8aha85TMOB8WwTZfgn35CA4KZoqb2i8/+/bn39i2bRUMbY21AmOPWrd9E+Jr3J+GuAfzC2lYHAletSCweFMr3uTMAoGdWJyNFGh3nygcO/FWm9odc7sjmF9Ig93ufk6wfgLQKxJY9HYpIqxGClQ7v8zp6bY/X8+VgRFsCEmwdKuZAk3ouGg34G56304wv3nLHrOkISK4/DJndPftuLs/bieYX1COnZ+OCD67zEU842PlV0e+n9aWYI0IForgLnbFbnf8FgQHnxDf/prDd4qYmU5EcJVlanTz3ToY49GC4N/xKidjZ560+TOz9UJKgnXoyo6IP4m356MFwTtERJSV4CUi3k8kJLibnfG025vKYAneICK+s3/DGGYe307wpOlULHEIzpC1vOPY2cXpxnPXguDZARFfLT4E+z0THBfzrqUkOD3dJ/c8X1KjI2ulBcHKv76+fs/hHfvm89vbs9czwXE1m5x7cGJH+HdMp2IVVpxQ38GNsQ3BJp9xe/eIiD8bgh3MJZA/nfevwNjL9WdeF5nTRzjKoAlu2Aq1rnHC8PxucyGc8pleu0Xcz/pWYBzX+MhxD463+S5qmloR7PG41P7dQAjms+4wX4fekwKtNLX+wMVlOP/72NTWh06wxSM1wv+KiIeppAQzd0HeqEAXAAD+k0sf89khKfvrJI2zFcHqzcb6Yj2rvMI+fH778g+KpAR/YH1y36hANeBnp+4xbpfuKoMnODbXb5DCBTcO08k7wyN4emDs4rhVgXrELdsdERFfGxeW90Lwqhq3S7eVf0FExB+Vb8Es+7H1Q/Ddblvtb5jcsU31vlmBsSnMI3UXERE/Rh3V47UjWKsSwmx/YXq5FreA/0PZ79g2x+iF4A+dtoW8XYFxzCxinl1ovSLi1wdTEYHgOEB5/heM7zDfuy9/i3hBRHx/sCs2dXZHGyeCJ8v1hSPm0GkL+BYKjF22Nx32083uw6TSPHl+/fhgK2IQbJd6/fQoNoUqTPl0MtK5FyMO5LDrxsaH4CvR2G5bwLdQYFw1d4vTdnKo9Akm9rWrCEJw6WALNYTEmC+1Z00AeHp+LNtrXcaRSD4EX4nGbjqNR7dR4CXapuvl5LIESi2l9K1QRSG4dLiQDfAJEd8fywN27qmBQuk1jmH1KB+C15d9uiW9KYeqwDgvoAzh5UWnfCXBiZOuy2rotgSXDHjTAABeXl8fKu4JIQCAV3IHTN5fbegEL3lmxHSrQKsK4T3ihbv4FLO9Kc8B1hVxCC4xI2I4Da/CyErGcobnrmSHeZtZTje5DeJhLgXBiczPEb5izC/2uJtVAdxtX6u2BJ+bEWnH+aqoTzKd3j6LSCcJpcrwCVZm88FUb7RVoFuO8P6GRvMpwJZQAjg3I44d5+MEqDNTOJlOrxV/mdgQhggED2i1VmA5wrMD4m4iAMDtBVA0I4yTKWCWuszTlEyvUIPicgjUE8G378JNk0u1fgBmIICCGZEd+uGU+Ba0NC3eyufHmzyKUojgJgi3kn3imet+/nh7ATg569XI3caC87uZdbQ6ouwDa1yKUojgJgi3ceKafQHMQAB67vrp5dyBCZd6EXjnKLcgZ0OxDkUSwY0QDm92IqRNbS0hBRBmrCitAOK5bRAcH9TIWNAun6IUIrgZwtFtBKrp/26KKYBVxi92NvnYLdwS1Ay2J5fwilOaFBFc/8/Ey7nBktCTO1ykCyoA7XR+aGe2QOIjM3N2cpiRWqQcI0Ps32AiuPay0kh/YwxXiQkcaMIKwDtaACXD5/X89mpnfGgp+vF/wmMmBxHc3JnQ8O9p6dg8TxVXAGbqRjDKJGDnXGp+1vfmATx8/Mu3f+SWj0cEN7Fm0wF6QQOPkH0b90MTQJgwWj4A2s8+Yc43YcSz1L5yy8cjghstJ00arOsaNsLkf4gMsQUQX8QMs/xlzLp6jXxfyjCelPc9r4TSvgleH3AzEYdgxUh31MiuoQ/jOHe3LwuCmQCSodNRRWpoxqVm54N0Vjwp7yOvjOieCV4jtxlUfAhWVA+ODF+5mJlHfqNVn0JmIwAb4OELvr9W+LTdYx9Ov7BL//YdEfGRV0p/zwTvOkgkthlboGZ4LEBwq51D2ir9z57ecL8Qn2A1jAe0v5QHJRKXmp2YwRmbSf2PH/D9hdshxI7g2Xx6I8F7sQhW1MzY09AxS/HNDE3/ARFxKjzBipkMaK9waif+sthSziUFqyvP5ncLYEbw5qbSt0UHPYht9l4Azc1NMrWtk4I0w/bCzC/DX/beZ5mVAJx3RMQv1sW7XuR3PI+PFcELvG2nWWx3S0UUgifzeTnD6UjeoPjvQivpNHonAcHKbxARf1L5axcA4On15aHT7A9WBK+x0y4mvShwdkDcH19SzYmujpm2UkvpMJVBAMryUq8lRQ0AXhDx6wOo4hG8xL6tPf4K3BYdJ5Z3Ad/QSVwVk/VuM5NCAFeNqwjeL7VBGbYdvL1+VM62u/VEYAWW1Haqllu6E/u2LuErfHUljXzedBEJVuZ3Vzaa6aHLTj0cFHhf7rzWLDtjAEe+uzKUQa3uCFb07xARf6UISXBNU3kqrgJnh0sGrWYYhq6sd7vBjYnskOC4/lWRmuCZwAqcLBZXrKAPiIiLEROsTOYda7hDguesB2QMUIHdduUcIsGdry6jyot73E7kVmC3XTmJYIWyKxmvzRCnRBLBRHB9K3Bz2K8VIpgIJgWSAIhgIpgIJgWSAIhgUiAJgAgmBZIAiGAimAgmBZIAiGBSIAmACCYFkgCIYCKYCCYFkgCIYFIgCYAIJoKJYFIgCYAIJgWSAIhgUiAJgAgmgolgUiAJgAgmBZIAiGBSIAmACCaCiWBSIAmACCYFkgCIYFIgCYAIJoKJYFIgCYAIJgWSAIhgUiAJgAgmgolgUiAJgAgmBZIA6izLIIKJYJGXVv0IEhCsEcHyCyAAX5OW4EAngqUXgAUQrWQl+JqCZDhER0+wCgAQGJJaEVB5wsiiQNNVR06wEo+w9jQpb3IBQGTLrcAo0EdOsJVM/XU1CQm2AABCU2YFOtDiIeS4CBwnsLu6dASryTx5Q14F6lVWYD2CXUP85R+Hr4NvSUZwYiNVMSzFFhSWnqB1CZZuRZIRbJ7eTlNSglex4uxbbnSWL8MKMwB7ZpZgGZ4u83DhSpWRYC3dfGxVGec6Ehw5Wt4Olm4FqowXmeCoQFcbI8B61StsyLDMjIF0rl8bQokP0bEsR+ojaAWXzlgZLzKr0REcAQC4sppQyQkbWuUmpBTnjJcx9a3x2cImALSJ6ohwy5H8jnO0InxDGeFypT53HAAAR+59SR81v4oaybsBx9uT1M8X70Ej5ldRLFfmDcqAESQARxdTP6RfcjsQ3VCXXoMmAIw2liG/kT8CzXry20m0pL7IgENCoCXyRcYiGeTX/wMciGAh9KpqDAAAAABJRU5ErkJggg==" } }, "cell_type": "markdown", "metadata": {}, "source": [ "# Feature Selection & Cross Validation\n", "A massive part of machine learning is an area called _feature extraction_ and _feature selection_.\n", "\n", "This involves creating new features from our training data. A simple way to create new features is by introducing **polynomial features** (seen below).\n", "\n", "## Polynomial Regression\n", "By getting the nth degree polynomials of the features for each sample, we can adapt linear regression to model non-linear behaviours.\n", "\n", "For example, in our example from earlier, the first chart shows linear combinations of the input, $x_1$. The second one shows how the model can be improved by including the square of that feature, $x_1^2$\n", "\n", "![over_underfitting.png](attachment:over_underfitting.png)\n", "\n", "For example, the linear regression model for a sample with two features $X_n = \\{x_1, x_2\\}$ is:\n", "\n", "$\\hat{y} = w_0 + (w_1*x_1) + (w_2*x_2)$\n", "\n", "The 2nd degree polynomial features are then:\n", "\n", "$P^2(X_n) = \\{x_1, x_2, x_1^2, x_1x_2, x_2^2, x_2x_1\\}$\n", "\n", "This would change our linear regression equation to be:\n", "\n", "$\\hat{y} = w_0 + (w_1*x_1) + (w_2*x_2) + (w_3*x_1^2) + (w_4*x_2^2) + (w_5*2x_1x_2) $\n", "\n", "Oftentimes, this can lead to better results.\n", "\n", "turning a base set of features to a higher degree polynomial is known as \"transforming\" the features\n", "\n", "---\n", "Below, we get the 2nd degree polynomial features for the first two samples in the training data.\n", "\n", "This is done with an instance of `PolynomialFeatures` in `sklearn`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:37:48.520691Z", "start_time": "2017-10-02T20:37:48.511666Z" }, "collapsed": true }, "outputs": [], "source": [ "base_features = X_train[0:2].values\n", "print('base features:\\n', base_features)\n", "\n", "poly_transformer = PolynomialFeatures(degree=2, include_bias=False)\n", "poly_features = poly_transformer.fit_transform(base_features)\n", "print('\\npoly features:\\n', poly_features)" ] }, { "attachments": { "cross_val.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABhgAAAMdCAYAAABp53yVAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAOwQAADsEBuJFr7QAAABl0RVh0U29mdHdhcmUAcGFpbnQubmV0IDQuMC4xNzNun2MAAMseSURBVHhe7P0N8HX/etf11RYpDzHinNCEQpp4BE090RM6EJnk2CYaE3KMEiTgURILY0hQITwYgSkQ4qQBJWMzAiatZIQpWiLU0A5SQpmW+oC0zSh2mBY7pGUsKna0oUpbbLFN1/U73+vO97/O2g+/vfe6f/u61us9857/fe/1uNfa1/d/7+uz11r/BQAAAAAAAAAA3pofRjnGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AAAAAAAAAADejtGzRiHGqQMAAAAAAAAA4O0YPWsUYpw6AACAMvyaxR9cjH/I/ND4+71842Ks73tf/vZ6Yj9i+Y++/A0AAAAA8GpGzxqFGKcOAACgBN++GP+AWXsvEVLEer7/5W+vJ/fjYy9/AwAAAAC8mtGzRiHGqQMAALib+PV//OPiu17+9ql8aPEHFmOeW64UiOWzkZ/biNc+/sk/3sVbBwwfXox9iO3nuuKqiDhOn1gEAAAAgPaMnjUKMU4dAADA3WRz/FSTPqdH4/yWWwlF8z6b7xEsPJK3DBgiQMhbLJ0y9uvR7xkAAAAAnorRs0YhxqkDAAC4m3MBQ1xxkM3yW684yBBgj3/AvFXAEOFCLhvPlYi/Z5AQ/42/5/MmYh8BAAAAoC2jZ41CjFMHAABwN6cChrmJHn++lW4BQ1zFkcvFraPOXaEQAY2AAQAAAEBrRs8ahRinDgBwBXGP9Gjy5T3k02hIfuOi25fg6GwFDHMT/dSzGa6lW8CQz6yI2yPF+AIAAAAAh2b0rFGIceoAvIJ1c/mU0TBCH+ZfYJ/y0i+Qr+HSfdjPGSEH8JasA4Zomudn+paHOifZiD/lujkf9RrLzPUU9fnti6ca+dcEDLHe+f8B8ee8IiNfuzZgiP3IZW4NXmLZ/H9N3HYqb6UU/10/4yL2K7aT8+R88dq5fY79XC8Xx/bUMnE88nOQ24hjK4AFAAAAcJHRs0YhxqkD8AqyaXKN6MH8C+wwmpTRXAujqT83Me/9hfa8ndfq9il4a+aAIRrK2Yy/N3y7NmCIbVyaN+p1qzl+KWCI2l6vK52nnWvWz8yh5a3PpMjl53Wl837kezvnVkAZxzXHt/hvHJs5YFmHGDE25rSYL+bP5eO8AAAAAMBZRs8ahRinDsAryOZJ/irzlM/4a/K5CbVuDOE0c/Nw67zOAUR8Lu4h1r/1ecrGbRj7szXP0c5pnpdoYOI5yM9p/Hc+P4/69Xp8zrMOtpgb3DHvHDzE+Dc3y3NakuuOfV8zbzfeV76faOLPtRleGzDM67z1+MzbjYZ+bjvGglxnhBc5TzT553Eips2BwTroyHMY73Hexzh28/aCeO3UemJc2zquAAAAAPABRs8ahRinDsAryAZKxWbJ3NC6tgmGDzYQTzE36fbAuftUrjkveL/M5ySNz+6jmOtgzdzgPrXNOQyMMGIm1701tuftgU6N+9mID28JGG4ll98KTJJL+x7BQc4T49hMns9rzmG879wfAAAAALiJ0bNGIcapA/AKsoFyqlnzzMwNLU3q68kmW7j1S+N4Lafv9blw7j6V+bzgOchzEv/NP0fz+1FX15xrysev5HPaVp0meQuldTM9172u4TmUOHcro5znLQKGuDpji2v3/dSxy+AkAohTAUZyTcADAAAAAGcZPWsUYpw6AK8gGyjrJlQFNKlvY/518lYj79L0R+DcfSrZwA7xHOQ5if9GozpvSRTN/HNN/5xv7fqWY+ea8hkcXBqbT60jX18vH/WW8597DznPWwQMp7Y5BwfnmN/jvK4IDeZzE2PduaBivkVVnLvY/qVgAgAAAADeMXrWKMQ4dQBeQTZPLjWxzhFN6GiGzY2baMBFw+lcAysaO7FcNvFmzy2ft784ZzSCZrJZF/89R76H9e1Ggmxu5bGKX9PO+x7Nqi1uPT5BNLNivfN7jmVj/25tdM2/zI19yn2I/+ZxCk+9n0cwNyNPNRNjf+J9xjFen/PY79jXc83P+dfOQaxvDk+2msRbx3vL2P4WsXzsc6x7nvfUvs7n4pxbv5iPz9X8+cvt7BUKHZE8vnPN5/GOY32KOBc532x8rmbONeXX2z7FqXXk6+vlz21zJuc5V2Mz19T0JS4tf+2+x/Kn1pU1ntPDOC+ngoaop/V4EMuvxw4AAAAA+BRGzxqFGKcOwCvIhsmlJtYW0WDJJtgpo9G21RwNtuZfG42daAjNnGrezUYjaubaZl0uvzXf3LCL97S1HzP3Hp9obG0tk156L+eYf5mbTen5/WwFLI/kmmbkPM851+c6mZuMcS7mpn86b/vS8V67Zn0Mt1yHNnPD+pzzfp56L7OnjileR9bvXGvzr+jXQeZrmT/ja7a2vcWpdeTr6+WjkZ7zn2uS5zzXfpbmz/Kt48elbZ47XjOx/KV1xXuP8zeHB1HDp4j1XAooAQAAAOADjJ41CjFOHYBXkM2SS02sLbIBFkbjJ4OAaLrMzdb47zokCOL1WEcsG02vaOCEsey87q2m7LrZE42iXD5cN35yfZfeZ65va75sbkVDKt9b/Dlej32JhtPM/B5ee3xinpwe651DiPhzbO/SezlHrH/9q9wwXjt3y5BHkccyjPO1RTb/4r2uz280MPP4hFufr5gvp2dDPpaJZcNYdy4X/53nnfdpPlc5bT4fwXpb8zGMeSPEyenr5mssG+Y+hvlaOjMfuzgu+VnPz1W8r/UyuI1T48Z8Pu851vO5XHNu2kzuy3ofc/n167G/ud5ztZ7zvOb95ZgS9ZKfy9dwaZvzvp/br6iLnO8a8hjG/l8i6jnXHfUGAAAAACcZPWsUYpw6AK8gGyXZOD3lulkUjZVcNpo5W8yNmFt+0ZrNqnCLa5rUyalG4Zpc39Z88/bCdfAxc+/xifeT0/Zo+EdDfW5oh/GeTzUFswEX/30Erzl3p5iPcaxvzXwMw3i/p95fHP+cbyusmLe1DheC/KzGfy8dw3BrnvyMhufI+eL9YF/yWK/Hgzh/9zbTg7kO1syf32vGkPU8ue71vgcZmJ2q5zm8fU19znVybnwMYv/W27+0zTjOl/Z9PjfXjld5rGLd15D7eeq8AAAAAMALo2eNQoxTB+AVZKPkkusGbjano5FzjmzQXZpvi7n5ttXAm6dfaoKdahSuyfVtzTdv71Lj6t7jMzcXHx0wRBMwm3Sz8dpW8zy49vhdy2vO3TlyHVsB1nwM4/ieawJfen+xbK5rXQtxfnJaHNtTzPuzNV/uQ3iOnE/AsD/nPhdzc//U5+YScx1sMX8mYt4Mv+LzONfx1mch1721b3OAEPNlbcRnNMeu9LX1Oe9zrCvqI9ef+x31GNPXIcQ125yPWYzD85gV28r9j2OzDgvjtVh+XmZ+z/O4HvPFe1nv/7z9U+MlAAAAALwwetYoxDh1AF5BNkouOf9SM5os+fqlKxPmZsxrmZfdajhdmj6TTa+tZttMrm9rvmu394jjM9+yJ5pi5xrXryHWk+uNJl80yOZ9OBUyZCPz0vG7ltecu3PkOrb2K9ab02N757j0+ZjPx3pd89UP2Yg8xal1BLkP4Tnm5nA0RLeuuMBjuPS5iHExz8WlWt9iroMt4vO0bvivjelbNZvr3tr3S+uN95Lv/bX1GeueP6OnjPWv6yWnXdrmpfXHeLUVzG7Nm67HvvncbLkORwAAAADgUxg9axRinDoAryCbJacaaFvMjdvXuEU0mKJJF9vPX7VuudVwmhtAlxpS2Sy79D5zfVvzXbu9Rx2fOQwI4/jEPtzaUI7lMiiI/87rmRt260bbueb6rbzm3MX02L88h1tuna/5PFza7/n9r5uewdxIXu/vuf065db+zOs5x1ZzOIKGrYYq7iOOaxzfc83kS5+dc+TnKmr7FPMYmdsJ4+/x+qlt5rrnX+XPxHLxOZzH3Zg3P9/53i/V5ymybuf1x9hy7rOa820FJmtifIx15ZgWRl1EOHJqjIx9imXmfYo/xzLr45jHZ33cz+0/AAAAAHyA0bNGIcapA/AKsmkSTZRriSZNLnetWw20aBDNzaFzbjW5XtOkzibRpfeZ69ua79rtPer4BLGudYMrjMbdqcbiKeYm+dYVEXOjdA4Z5qDj0nG+lmuOZby/rfe+5db5ms9DbO8cc4iy/kX4/DndOk/X7uPs1vGf13OJbH6u6yf2T/MTAAAAAIAnYPSsUYhx6gC8gmxMbjVoT/Gaxu0pooGb64gmaTS/1786vdSEvqZJnWTz9tL7zPVtzXft9h5xfNbE8Ypf2c4N5WiEvyZkmBvYp/Z/K2SIpnX8/VQIcgvXHMv8BXV46lfDOX3rfL3mPMRnb92sX5vHY818XO/hlvXE+Y+wYl423AowAAAAAADAe2T0rFGIceoAvIJsSG41aE8RTc1cLhq/tzA3sk81mC81oa9t+AfZgI2m/DlyfVvH49rtPeL4nCLWPR+71zSS5yb0uf2f1z/7yKb1pWM5X1Fw7rN5bp5Yb06/FDDke47zFfNmqBLGZyZeOxXmRPCT824FENcyn59biPebIckjwyAAAAAAAHADo2eNQoxTB+AVZEPzXBN3i2zARkPzVOP1HNc0Uy81oS9Nn5m3d2p/56sqto7Ha7Z37/E5R6wv9yP26VpeE0zMTfMw3scjuXQs47Wcfu495jxb5+vadQQ536XzukUcy2u3c45rauIS83kDAAAAAABvyOhZoxDj1AF4BdmM3GrQnmNuEEfj+hzxa/R1k31upq5vjRREsz9/jR1uNX7jljk5PW6xdI658bo1bzSJ5+3dGzDce3ximdjnrXDiNY3zmbkRHu/11K/tY5tzGJHGr/u39ucWLh3L+T2eOn7zPj4qYIh1vvY9xvz52Tl3XJNT06+5EiI+KxFenfr8PSKkAAAAAAAAD2D0rFGIceoAvIJsRr42YIim6nwbmfhzNO6j8ZlGUzcbnusG79zsnh+qG+uNeedmf7jVUJ1voxPzxzpzu+v79cfr8/qimRuvxXzROJ/XE/+9N2C49/jk6+vl4s/zsdkKZ86R6w1jPXEc4hjEuuO/8fd5/XFuwvx77E8c59dud801x3I+fvG+s/Ef88/vI9w6XzFfTl8f3zXr9c3G+4/p8TmJ975F7N+8TAQVeVzDWC5ey/e0xbomcvlYLo/3/J7mbcR/589xTAMAAAAAAG/I6FmjEOPUAXgF2ZDcatBeIkKBufl8zmjArrm0bDRJ88/RRN1ibqrObjWU51+Ir42mejRps9G8dTyuaYrP3HN8Yl+25ps91ew+RzTpzzXTZ+P9xvzh+jhfathf4ppjeekYxLHN47t1vuZm/KX9XQdC54xtZtgxM7+nc8ZnbYtz+zAfo0ufqVP7BwAAAAAA3iOjZ41CjFMH4BXkr9WjgXwr0eiO5edfvsefo+mbVwqcIhrrc8M0Gqz5i+1o0Ofrp24ZE43UaOzOjdnY7qn55xAhjOViH7Mhm430reORv1KP9/aaX/DfenxiG3Es5vc2H597iG2u1537FO9zq0Edxy7n3wqMXsN8LE+dqyCmxX7Oxy72McOVc+dr/vzEvp9jDp9i3+L4pLGt9Wcs9mmLOC+xrnUIEH+PfTx1bJM85/l+47+x3HqZ2Kf5cxzOxwUAAADAkxD9Kh5LIFk+DyjGOHUAAFxNhAjZpD8XnESTP0OG+C8AAAAAXCT6VTyWQLJ8HlCMceoAALia+eqFS+RVA3FlAQAAAABcJPpVPJZAsnweUIxx6gAAuJq4JVEGDJduX5Tzbd2SCQAAAAA+hehX8VgCyfJ5QDHGqQMA4GriuQUZHMSzEuLvc9AQwUK8Nj+D4dxzIwAAAADgHdGv4rEEkuXzgGKMUwcAwKuYr2I4Z9wa6dIDowEAAADgHdGv4rEEkuXzgGKMUwcAwKuJhz1H0BBXMaxDhXj2QjwA+twtlAAAAADgU4h+FY8lkCyfBxRjnDoAAAAAAADg7Yl+FY8lkCyfBxRjnDoAAICSxAPE4x80ccXMI9lrvQAAALhA9Kt4LIFk+TygGOPUAQAAlCRuxxX/oPk1L397HHutFwAAABeIfhWPJZAsnwcUY5w6AACAm8lf+8fzOK4lG/jxHI97OBUEfHgx9iumfzReeCVvFTDEs01i27HvnmHSlPg3ON9GAEANtsZw9hZIls8DijFOHQAAwM3EbYTiHxXhNc38aP7n/J+IF+7gVBAQjfrcxi0hwVsFDLG93O94D2hI/BucbyOOzdZngm8jcImtzw17CyTL5wHFGKcOAADgZuKX9vGPivDb44ULZCDxQ4v3/kr/VBAQQUesP6Z9PF54JXsEDNc81yEClzw2EcSgIfFv8Gtc+Orhj9uazg+68Pcvfu3iR7emhzg2W58Jvo3AJbY+N+wtkCyfBxRjnDoAAIC7yOb5D7787TxxK6WY997bIwV7XWmwx3rf6qoIPBnxb/BLLvzcmHX4C7fm4Sdd+OmLf3Qcq/Cf2povxLHZ+kxUdKF8+PgeiJD+7138tYv/yOKXLv5XFlGErc8Newsky+cBxRinDgAA4C7yl/fhuVv7zLdHuuXKgjUCBpQj/g1+yYU5YPi6rXn4cpy+dTpO6XduzRvi2Gx9Jqq50CJ83JG/bzF/yLD2/7P4TYsowNbnhvEx7nt14wP5Gxd/4eKvW/zVizFufmQRRVg+DyjGOHUAAAB3k7ckOndlQt4eaX2lQwQPcXulbMKn8fdztxQ617TPdZx6LkQEIXnlRRj7FOuJ2zadW+9r9zWPyynn/ctnR8Qyp4h54hjH/uY64s/x2rlwZ15vvIeYP/ctl/dg6fdA/Bv8kgtzE/G/vTXPkV34Oxb/rXF8wv9w+vM/vbVMiGOz9Zmo5kKL8HEnfsNiHptz/tOLeHK2PjdHd6FFwHjKB/CLFv/UYh6jtd+x+KMW8eQsnwcUY5w6AACAu4kGdfzj4lxzPH9VOD+rIRrs+Q//U8a6t7gmYNhqus9XXKyNfcz9XK/3ln29JWAIt4j9mZfd8lQgk9MvvXchw87Ev8EvuTA3EX7x1jxHdeHHLP7n49j8e4tfsxi/Vszj9du2lgtxbLY+E9VcaBE+7sRvXYyV/4uLP3sxxorgxy3+ssU8bmE8swVPzNbn5ugutAgYT3kncaVCHptzfs8inpzl84BijFMHAABwN3HLo/zH+9btj+bm/LqpHr+gj9BhDgPyV/ZbyyS3BAxzEz8a6rGdYL29cL3evfY1mfdtzXx848qLeRsxLUORcOv457QwQo8IGoIIFOL95LRzV4zgAcS/wS+5MDcR/sGtea51Ic7xZ2xNu9aFn7T46VvTXuPCT178L21Nu9aFaBrGH37V9NocMPyOef5ZHJutz0Q1F1qEjzvxdYt/9yf/uMk/upjH7rfHC3hetj43R3ehRcB4yjv5ZxdjJf/E4vp5K//AYh63MAJIPDHL5wHFGKcOAADgIeRte7auOMgmdszzGvIKgK3G/C0BQwYBsd6tX+vPv/A/FwZsceu+JucChjy2sZ4t4r3kPBE2rMn1zqHKTAYUp9aPBxH/Br/kwtxE+Iatec65EL/O/Z8v/geLuZ74czwM+RdtLbN24Zcv/q8W/6PFXMe/u/iHFv+urWXChV+5+J8s/vvj73/V4j+zmPvy/1v81xa/bb3srS7MAcN3bc0T4thsfSaqudAifNzgJXz85B9343MW89jFWIgnZutzc3QXWgSMp7yTCBf/uk/+cZPfsZjHLv6dgidm+TygGOPUAQAAPIQMEbaa99n8fu0v5M815m8JGDIEiH09xTVhwBa37mtyKmCYr/7YujohiWOb862P/6XlY79i+lY4gQcS/wa/5MLcRPilW/OcciFvE3LO37W4+YDIhb9i8Z9bXC+z9p86sfwvHdP/08UvXvwz4+9bnryd0WtcmAOG//7WPCGOzdZnopoLLcLH+MvCyfBxcQ9+7GLu778UL+B52foMHd2FFgHj2oX3ETDGA+Dz2H13vIDnZflcoBjj1AG4gvnWF3lbiXPMt5vYulXGTEzPxlM0vt6K+EVr7nM0mNbNqTUxfzbq4nYdW8SximnZWAzjz3E8t35Bi9PscX6ARzM3wuexcn79VO3H69Egj/EhxsQ0P8evbdrn9tYBQ75+rlF/br3Bo/c1ORUwzMHBOeblT73v9etJ7FfOgx2Jf4NfcmFuIvzDW/NsufCd03L/y8WvWvyMxfjyHr/u+xOLOX2zub/w+6d54l7m8UDlaATG5/7nL/6fFnP6r9lYPgOG2X9h8asX/2uLv3gxnp2Q075xvY7XujAHDP/s1jwhjs3WZ6KaC4cJHxcfzRcs5vo96PnJ2foMHd2FFgHj+Pv7Dhhj+7m/vy5ewPOyfEZQjHHqAFzB3ByLBtK55u7c4Dn3S9BYxxxEpG9JNKJzP841wYJsloXrRl0crzlUOOU1YQ1+hEedH2BP8lY7c7CVY118LrfYGgvXvrZpn8vNDfVzDfiZc+vdY1+Tef9mYpmt19ece3+nXk+u3QbuJP4NfsmFVwcMC3/TtMyfOTHPpy/+62Oe8COr6V85Tfut87Rpnp+2mCFDNAv+6tX0dcDwKU2Lhb91mv4n1tNf68IcMHzP1jwhjs3WZ6KaC4cKHxcfyfwQ2L8rXsDzsv78MD62AsY7+IOLuf6/PV7A87J8ZlCMceoAXMl8FUM0mE4xN3ZPNXKisZ6/dF37lsQXhNyP2L/4+xZzE2urYTgfg2g2RuMqnI9hGmEEruNR5wfYk63b9GTguBUqzo3t+PN6TDjXmD83Ldc5j8OxP/n6LVcw7LWvyVy7M/N2zzEvP7/v4NTrybXbwJ3Ev8EvuXBLwPA/npb5/K15woWow5zvA42Ahf/1eP3/ufiB4GB2Yf68/PLVtLmJ+N+Zp80uzKH5F2zNc60Lc8Dwz23NE+LYbH0mqrnQKnxcXPOB8DFeeCD/1mKs9z9b3Pt2LLiT+bPDT7ogYLyN+d8b/7N4Ac/N8plBMcapA3Al0ZiaQ4Gtxvj8pX3+9e7M/OvXaLrFMtl4Ct+auWlw6j3MVydsHYdY7tRtkGL++TjGfLieR5wfYE/mICzGt/gM5t+3rv7Kz+upseDWpn1u81Sj/ZqgeL3evfY1ORUwnAsOZs7dSunS8rFfp5bFA4l/g19y4ZYm4r8z5v/+remzC/n/4T+wev0vjddPPscgXPgpY77wO1fT5ibC583TZhf+nmm+k7dVuMaFOWD43VvzhDg2W5+Jai60CR/jhRPMzcC4rdEj+JbFXOevihfw3MyfHX7ShVYB4zx9zPOIgPFXLP6ji3EbpPj3+r+9mOuMsRAFWD4PKMY4dQBewdy8iWbRTDTOsvl07tflsVzMF/+IT7Lx9AyFOb+PcN2Mmo/BuQbdOeYvMLEtXM/7OD/AvWRzIMa2DFVPBWL5ed1qvM+f99c27XO96xrJfYv1bgUe8/i0Xu+p14Nr9/VcqBr7mtuYiXVnQ/jUcZy3vzVPrlfA8MbEv8EvuXBLE+H/PeY/2WBPF/4PY95/e3otfskYfwi/dZ5/y4VoNMYf/ier168NGL5omu8f2prnWhfmgOF/uDVPiGOz9Zmo5kKb8DFeOMEHwsd44U7mpmXcax4FmD87/KQLbQLGedo0T5D/Tg9fGzDGQ6dz2dn/fPGfWPzcRRRg+TygGOPUAXgleX/xcL7Fxvw/1K3mUrIVPDxTwBDE+8r9mZ8jMTe54r9bzblrmJtoIV7H3ucHuJd1iBj/PXVLohz/1g3/uPJhHm/PNe23puVy64b6vG/xJSbH5Pjv/KUmXK/3nn2NYCGmRW3mlUVxTOb1nAoYglhnTov9nK9OivXk9mP9W/+fyWUFDG9M/Bv8kguvaiIsfPY0/7+xNc/swv9izPsfT6/NTbh/fp5/y4X8peK/uXr92oDh86f5fsPWPNe6IGDARbY+E9VcaBM+xgsn+ED4GC/cQfyq+t9fjHXF/x9j3EEB5s8OP+lCm4BxnjbNE9wbMMbDpr9v8Q8s5npmz109hSdh+TygGOPUAXglcwMomkxBNHPyf8j52mt4toAhmPcpGnLBfHunfO0Wto4hXsc95yevcoh1BNHgjOZinItcPpqVl87x/LnfaqjeQqwz9+HSFRh7bB+PIT5TeR7DOE+nmMeDmC8+l3OzPv+8dY6zDram5fJbDfVs9m8Z68xaWq/3nn2dl52d92+eZ4tz+x3GPp0KcnKereMRxD7nPNiR+Df4JRdeGzD8+Gn+a5qI/8cx7w9Mr31kvBZe00TMpuXvX71+bcDwt03zff3WPNe6IGDARbY+E9VcaBM+xgsn+ED4GC/cSNwf/t9czHX97EUUYf7s8JMutAkY52nTPMEjA8bk71+cnzPxHYt4YpbPA4oxTh2AG5h/5RpNmbnpc6p5c45skoXPwnzf9GhazX+P/b2HaFznuuJY4vXcc36ykRjzxec1ls9l1567ncvckHxkUDQHHfOvu9fMt4M61VTF2zEHXpfCovj8xliQn8X4b3z24vU8z/HfNTkWb03Lbcc6tohxaB0OZKiW29wKCm7d12C+0iCM9cxXG2TAEOs8RezjvP0w1hnHeF7Xmpz31PHIfT+3bTyA+Df4JRduaSL8h2P+P7I1fXbh/zvm/X3Taz92vBb+D+b51y7EbQZy3g/cj3nh2oDh66b5vmprnmtdEDDgIlufiWoutAkf44UTfCB8jBdu4EcvZlgS/n2LKMT82eEnXWgTMM7TpnmCRwWMa+I75Z9azHXHOIMnZfk8oBjj1AG4gfmX03OT59bG+zMGDMHcIJzf5y0hykw2BcNLv5LHaW49PxkMrD+70fyMZWP6NevbK2CY13vu85GN2thXAChB/Bv8kgtfHbMOrw0Y/o1pmc/Zmidc+AXTfP/Eatp/MF6PcfWvmKfNLnzzmC/8patpcxPh58/TZhfm2xf8jK15rnVBwICLbH0mqrnQJnyMF07wgfAxXngl/8XF+bvVrSEF3pD5s8NPutAmYJynTfMEjwgYT/HLF3Pd8TBoPCnL5wHFGKcOwI3MTdD03K9Hz/GsAUMk/XOjOTz3i/ZryF/ohrHuc79Qx3luPT/rz+7Wr62jsZ/TIxDaIn4JnduPdT6KqKPc9qnQbp7n3s8kALw34t/gl1y4JWD4R6Zl/tWtecKFPzbm+X8sfsZqWjwEMdfxTfO0aZ6/ZjGvNPt3NqbPTYTw523M87Om6f+b9fTXuhDNylyfgAGbbH0mqrmwZ/j4907z7R4+xgsn+ED4GC+8kn95MZf/1ngB9Zg/O/ykC62vbhzcGzCeY96H3xQv4DlZPg8oxjh1AG4kmqv5P6gwfk19K88aMATz1QbhvbeimW8P4rY293PL+ZkDhlO3cgkyPHjk1QnXMtfEVnA3v4dTt3wBgKcj/g1+yYW5ifg7F//uxa/ccrXcfPn/71j86DTtCxb/1cWc/o/Ny455Pm3xL4zpYXwB/xum6fHFP+/RHP598/JjnnXAEMYvBT9j8Uctftnin13MaX//eh2XXPicxZ86+Xcs5vqiORnT079mWg4HZv4MVXWhVfi4uOYD4WO88ErmcCLGThRl/flhfKT7BIzztGme4N6A8Rz/2GKuWx/iiVk+DyjGOHUAbmTd2A1vvd3PswYM6xAlvKfZPN/S59Sv4nE9t56fuTl/7nZKb/m5nK+g2Lo6Ir/AvkX4AQA3E/8Gv+TC3ES45E+dlosvzP/n8Xr6Hy/mQxfTb5+3N7vwtYv/9zFf+u8t/uer1371ieXnJsIfmv685T+ztY5LLvzfpnVc8l+elsOBmT9DVV04VPi4+Bq+ZzGX/d8uxlUSMSbGw523/CmLeFLWnx/Gx7pXwLgxzz0B4y8e/z1F3Drtzyzm+n/iIp6U5fOAYoxTB+AG5tv8RBM2f+kd/73llj/PGjDM+zUHKrfcDmduGMdVDG6NdD+3np8KAUNw6gqKOVg5dwUGADwd8W/wSy5E4+v/uhjBQBhf+Nf+xeFPXi374xb/ucUIFnKsDP/TxWggfKDxuOVCPBjyf7r4nyzO6/iPFv/I4hduLRcuzE2Ev2Exmnxz4zH89xe/eWv5a1x4TcDwB6flcGDmz1BVFw4TPi6+lv9scWtdp/wfLeJJ2foMHd0FAeM2cSVjLPN/WYwfNMZ3xQg8gug5/J2L85Ucbo/05CyfGRRjnDoAN5DpfAYKc8P2lub7MwYM61+Qz/f7f22QMq/r1hAGH+Se81MlYIhnK+T259sgzVfC3PrcEwB4E+Lf4O/LhZ+8GL/U3bzf8TUuxD2V/67Fv3Zr+tqFDwQM0+tfvBjr+a/O89/qQjQkLrpaBgdm/ixUdeEQ4ePiLfzpxf/X5LzeLX/bIp6Urc/Q0V0QMG4TD7OeA5BzxvdbPDnLZwbFGKcOwCvZChOimZuhQ/japuezBQynmtXxa/Hcz2sfrLsOF+ZGMW7j3vNTJWCYr1SY30/WmttsAShH/Bu8swubAcMziGOz9Zk4qgtvHj7GC4N34ePL34CFrc/Q0V0QMJ7mb1yM0PDfXZzXGUYI8scX44oPFGD5zKAY49QBeAURHGRjN5qc2dgN5kb6a5PxZwsY5l+Ir29BMwcp55rTwdwgFi48jnvPT5WAIciHgsfnJ4j9zX2KmgOAUsS/wTu7IGDAU7L1meD7c+FUwAB8ClufIT7Ghae8ujH+8iD+5sX4zvjli652L8jy2UAxxqkD8Arm+9xvNTezGRpear7PPFPAMIcC0axeE5dY5vRzQUqsZ/6VvXDhMTzi/FQKGOarMuK95W2T4jM1B3wAUIL4N3hnF/6heJtDAQOehq3PBN+fCwIGXM3WZ4h1Xbj44wMgWT4PKMY4dQCuZP7l9KnG7TzPVvP3FK9t5M5XS0TT9ZHM+3Kq+TzPsxW0PCpcyF/jR3DzaDIsim2sG9XZ1I59Xx+D+Hu+t7xF1pq5Cf7oX0084vw8KmCI45bn6NpbZr2W2EbuQ5yzPPZ7bQ8AdiX+Dd7ZBQEDnpKtzwTfnwsfGBviBeAUW58h1nVBwICrWT4PKMY4dQCuJBup4bVN2a0GdDRMY/nZ9ZUPs1u/0p4bxKea3LcwBxfxPk4R+5XzRcN3vY/ZBA6jKRz7eM5TIUmuI3w087rj/cycO4fx95x26hjNy6/XfQ+POj/zezi3f/P72GK9nb3IwGb2kccVAN4b8W/wzi7E/9P/hcV/fvEhD3R+lDg2W58Jvj8XBAy4mq3PEOu6cPHHB0CyfB5QjHHqAFzBtQ/PDeIX6zlvNF3Xv2Cfm7vXuL7HfjCvY+sX6rcQTeg5RLn0y/u56RvPBEjmpvO1nmpO5/TYr0czb3/drH5kwPCoKxgedX6CPQKGU8fiEcy3fQr3+DwAwHsh/g3OtxHHZuszwffnwgfCx3gBwGEQMOJqlv9noBjj1AG4gmzsbgUGW5x7CO8cVlzjVoCQjd+tX6ffyvzr+HVDeos4DrH93I8k9idfv9a4ymHN3LxeN/kfwV63SJrf/yOb7o86P8H8/s59nvMYrZdP5sb/HudoZg5X9t4WAOxG/BucbyOOzdZngm8jgMMhYMTVLP+fQDHGqQNQjLnxvnV1QxcyRNkKAJ6Z+eqA7g+2znO0xzMy1sxXZFwT8gHAUxL/BufbiGOz9Zng2wgAwCmW/0+gGOPUAShGNnUf+ev4Z2MOUdZXEDwz89UL3X9lP5+jvYOUOK65rfcRZgDAbsS/wfk24thsfSb4NgIAcIrl/xMoxjh1AAqRTd1oYlf6Vf9ryRClWpM+r17oHP4k7/MczbeH6nzVDoADEP8G59uIY7P1meDbCADAKZb/T6AY49QBKETeEz/uYdiV+DV8vMeKv1SP4OfScw06kOdo69kZexCfhdhe6PZIAEoT/wbn24hjs/WZ4NsIAMAplv9PoBjj1AEA8LRkmBG+r0ADAHYj/g3OtxEAAADPzfJvNhRjnDoAAJ6WuCVS/A8rrgzpfOUOgIMQ/wbn2wgAAIDnZvk3G4oxTh0AAACA90H8G5xvIwAAAJ6b5d9sKMY4dQAAAAAAAAAAvB2jZ41CjFMHAAAAAAAAAMDbMXrWKMQ4dQAAAAAAAAAAvB2jZ41CjFMHAAAAAAAAAMDbMXrWKMQ4dQAAAAAAAAAAvB2jZ41CjFMHAAAAAAAAAMDbMXrWKMQ4dQAAAAAAAAAAvB2jZ41CjFMHHI+P/j3f+cMkezvKHcCD+dJf9T0/TLK3o9wBAADeK6NnjUKMUwccj61mJMlejnIH3pqfsPi5w48tfsniVy3+ouFvWPzWxd+y+LuGf3jxj234Jxb/7An/0mJ87sNY325sNSNJ9nKUOwAAwHtl9KxRiHHqgOORDcgv+9X/OslmChiwM5+1+PmLX7YYAcE3L2Y48EcXf2AxGv7Z7H8L30vA8At/5/+eZDP3Dhhy/ST7Osp9F776W/4bP0yyt6jHGKKB4yFgIPsqYMAdxNUGP2PxaxYjOPjuxbiaIEKDP7cYn6sKChhI3uTeDcJcP8m+jnLfha1mJMleoh5jiAaOh4CB7KuAARf4tMUIET6xGI3437sYAcJfWIzPTQcFDCRvcu8GofGD7Ove40eQDch/8X/33yPZTAFDXcYQDRwPAQPZVwEDBj9qMW5lFFcjxC2M/uDiW9+66H0pYCB5k3s3CI0fZF/3Hj8CAQPZVwFDXcYQDRwPAQPZVwHDYYnbG8XDk79tMZ6F8BcX43Pw1v7lxXwQc1wpEQ9rjv3LBzp/x2IEAvGw53zwc7yPeBj02nhIdD4weu2PWXwvaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYDkNcoRAN9wgUonEf5/x9+B8t/qnFeDZDBASx/V+3GMHAz17MECBuxdQSDUKyr3s3CI0fZF/3Hj8CAQPZVwFDXcYQDRwPAQPZVwFDa+JX+l+9+HsW93hmwl9a/NOLER7EA54jOIhnNcTVA+/1KoFnRoOQ7OveDULjB9nXvcePQMBA9lXAUJcxRAPHQ8BA9lXA0JIvWIxQ4VG3PYoQIZ7JELcn+qWLcdXBZyziCjQIyb7u3SA0fpB93Xv8CAQMZF8FDHUZQzQOxocXf2gxPgBxS4lDImAg+ypgaEU0/uNqgjiftxiBxJ9Y/O2LEST8jMW2ty56X2gQkn3du0Fo/CD7uvf4EQgYyL4KGOoyhmgcjGjWZOPlsB8CAQPZVwFDC+KBzb9zcf7/1TXGlQnxXIRfthhXPcRzGvBgNAjJvu7dIDR+kH3de/wIBAxkXwUMdRlDdBk+vvhdiz+4mI2E8PsXv30xGue4TJWA4RsX80qLOOcPRcBA9lXAUJ7PW/zzi/P/q04Zz2H4vYtfvxjPSMB7QIOQ7OveDULjB9nXvcePQMBA9lXAUJcxRD89H1qMECGbCeeMoAHnefaA4dT5figCBrKvAobS/JTFS+FC3PYonsfwZYuuUHgDNAjJvu7dIDR+kH3de/wIBAxkXwUMdRlD9FMTzeZ4TkA2FcLvXfw1w/UVDdGYxnmeOWCIq1TyqoW1D0XAQPZVwFCaP7a4Hv/TP7MYz1H4MYt4QzQIyb7u3SA0fpB93Xv8CAQMZF8FDHUZQ/RTEyFCNhai8fzRxS2iaR7hQgQOOM8zBgwRJEVwNO/XOlh6KAIGsq8ChrJ8xuJfXpzH/jBug/SLFl2t8CRoEJJ93btBaPwg+7r3+BEIGMi+ChjqMobop2a+OuET8QLu5hkDhvk8x5/jSoZd91PAQPZVwFCWr12cx/0wHtoct03CE6FBSPZ17wah8YPs697jRyBgIPsqYKjLGKKfmmwwhNFwxv08Y8CQ+xLP0IirGQIBA8mbFDCU5VsX53E/jNfwZGgQkn3du0Fo/CD7uvf4EQgYyL4KGOoyhuinJhsM4b0BQ9xeKW65FLdSWt/nP177xsVsbp8ib9sTt/NJYr/m2/vEuuNWTVv7G1dhzPPGr/Vj3nPbjf3O+WMfk9zu/F7i7/Hr/3O8tnH/4cXYx/mWRbHNOGaPuqpk63i9dj9fhYCB7KuAoSwChiJoEJJ93btBaPwg+7r3+BEIGMi+ChjqMobop2bdPL+VaIbnes4Z2zv1nIcg54v1RSgQjfF5+bXZgI95z+3Due3OjfYISGJdc0ixZezXKV7TuI8rCuZ5t4zg4dwxuxUBA8mbFDCURcBQBA1Csq97NwiNH2Rf9x4/AgED2VcBQ13GEP3UrBv45xrnp1g3qqMhHo3zaNaH62Z9XFVwinkd8y/6IzyIdcX+zaFIGFcUzPPG9nLeeb5T2533P/Z7Xlf8Od/H/ByDnHeLaxv36/2btxXvYX6fse1LV3+8FgEDyZsUMJRFwFAEDUKyr3s3CI0fZF/3Hj8CAQPZVwFDXcYQ/dTE7XnWDftoZr/m1jy5jmi4x5+3mG9DFJ66zdA8TxjrXc8b61rvc867/qX/ertb72vdaA9jXfH6mnUosPV+r2ncx37k9FPbinXPYcct4c85BAwkb1LAUBYBQxE0CMm+7t0gNH6Qfd17/AgEDGRfBQx1GUP003OqYZ9Bw6N+OT835+NX+lvM298KDJL1rYXOzTtvd6tJv260n1tXMF/JsHUVwzWN+3kd557pECHDvK5HXsUgYCB5kwKGsggYiqBBSPZ17wah8YPs697jRyBgIPsqYKjLGKJLEI3sU88wiIb7/PDjW4lQIdd56nkP83bPbXPdHD8XCMxXC8R7XPOadQVzuBFBwZpLjfsIFM4tv2a+xdS5MOK1CBhI3qSAoSwChiJoEJJ93btBaPwg+7r3+BEIGMi+ChjqMoboUkTT+VTQELfruecX9HPAsNXoD+btxb6cIvZjnvcccyP9moDhEpfmvzR9Pg6nnuMwM89/6sqPW3jt+34VAgayrwKGsggYiqBBSPZ17wah8YPs697jRyBgIPsqYKjLGKJLElc0rJ83EF4TMsT0aF5HM3x2Di7uDRiCed5zzI30RwQM69sWra94uLS+9XFYH6e18/zx90fx2vf9KgQMZF8FDGURMBRBg5Ds694NQuMH2de9x49AwED2VcBQlzFElyaa5/ODhsNTTe64fc963lNWDhiCef71fl5a3xwYvFYBA8k3V8BQFgFDETQIyb7u3SA0fpB93Xv8CAQMZF8FDHUZQ3R54oqE+aHEW88N2Lra4ZwChtv0DAaSb66AoSwChiJoEJJ93btBaPwg+7r3+BEIGMi+ChjqMoboFswPSl6/sXgY8zwtHkocjfD1rZTi1/c5T+WAYX2LpPj7zGsChkdekfBaBAwkb1LAUBYBQxE0CMm+7t0gNH6Qfd17/AgEDGRfBQx1GUN0C841o+erG841zLsEDBGenJv/0vriwc457ZqHPO/Fa9/3qxAwkH0VMJRFwFAEDUKyr3s3CI0fZF/3Hj8CAQPZVwFDXcYQ3YJzVzDMr69/zT9TJWBYP7R5zXw7qFsCi/lYxjMr3goBA8mbFDCURcBQBA1Csq97NwiNH2Rf9x4/AgED2VcBQ13GEP20RKM8GuTnQoEgbnU0P7w5boE0k6+H55rzsVzO98wBQ7zX9e2dknj9hxZz3rg91JpLjftYxzz90vvci0v7eRcCBrKvAoayCBiKoEFI9nXvBqHxg+zr3uNHIGAg+ypgqMsYop+WaLZngyFu1bMVNEQTeg4XwnVDfL5FUoQWa2K987bCZw4YwnhP67BkHbRE0LAVRKzXt8V8m6RYz6WHN8f0uPLhkVyzn1vM+35yvwUMZF8FDGURMBRBg5Ds694NQuMH2de9x49AwED2VcBQlzFEPy3zLYvSaKxHEz6cg4N0q8m9vn1SLBfrDuerFmafOWCYr1CI/Y/3EU31+fXwVHP9msb9OqwI47jFdvLYxZ9jf3N6bP9WYl9zvel8q6dwPf1UoDEvs76a5R0CBrKvAoayCBiKoEFI9nXvBqHxg+zr3uNHIGAg+ypgqMsYop+WaHLPv0Y/ZzS3z/2Cft2sXpu/0s+/P3PAEFcurJv/a88di2sChiCO/xwgXPJkM/8K1uHItW4xT4/PzyYCBrKvAoayCBiKoEFI9nXvBqHxg+zr3uNHIGAg+ypgqMsYop+euIVRPEsgmt1zIzr+HK/FtK1bAa2JACGa4PM6olEfv4bP5XPa1q2Ugmzsx3yXng2R64pf/p8jAoOcd6tJvxUIxP7Gfs9BQ2wn9vvSfsX03F4sf4nYfqx3HWrEOuL4RxP/5K2IruQ1QUZ6at/nebaeQfGCgIHsq4ChLAKGImgQkn3du0Fo/CD7uvf4EQgYyL4KGOoyhmg8OddecYBPhid5nCIAORk8CRjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4jGkyNguJ75VlhxhcdJBAxkXwUMZREwFEGDkOzr3g1C4wfZ173Hj0DAQPZVwFCXMUTjyREwXEdcrXD1rZ8EDGRfBQxlETAUQYOQ7OveDULjB9nXvcePQMBA9lXAUJcxROPJETBcR1yxkMcojtlZBAxkXwUMZREwFEGDkOzr3g1C4wfZ173Hj0DAQPZVwFCXMUTjyREwXGa+eiEeOn0RAQPZVwFDWQQMRdAgJPu6d4PQ+EH2de/xIxAwkH0VMNRlDNF4cuLBxVff+uegfHwxj8/JBzvPCBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4gGjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKGaOB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgezrawOGMSzgDRin4B3jZeB4CBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEO4xS8Y7wMHA8BA9lXAUNZBAxF0CAk+7p3g9D4QfZ17/EjEDCQfRUw1GGcgneMl4HjIWAg+ypgKIuAoQgahGRf924QGj/Ivu49fgQCBrKvAoY6jFPwjvEycDwEDGRfBQxlETAUQYOQ7OveDULjB9nXvcePQMBA9lXAUIdxCt4xXgaOh4CB7KuAoSwChiJoEJJ93btBaPwg+7r3+BEIGMi+ChjqME7BO8bLwPEQMJB9FTCURcBQBA1Csq97NwiNH2Rf9x4/AgED2VcBQx3GKXjHeBk4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCowzgF7xgvA8dDwED2VcBQFgFDETQIyb7u3SA0fpB93Xv8CAQMZF8FDHUYp+Ad42XgeAgYyL4KGMoiYCiCBiHZ170bhMYPsq97jx+BgIHsq4ChDuMUvGO8DBwPAQPZVwFDWQQMRdAgJPu6d4PQ+EH2de/xIxAwkH09SMDw0cXY9x96+dvrieVi+VjPmzFOwTvGy8DxEDCQfRUwlEXAUAQNQrKvezcIjR9kX/cePwIBA9nXgwQMH1vM77q3kMvGet6McQreMV4GjoeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvr5RwJBXFFzyUVcMPEPAEMv+4GKs59fEC69lnIJ3jJeB4yFgIPsqYCiLgKEIGoRkX/duEBo/yL7uPX4EAgayr28UMMwN/3N2CBg+tPhdi7mOUMAA3IOAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvj5BwHBL0/61vFXA8PHFfH7DrIABuAcBA9lXAUNZBAxF0CAk+7p3g9D4QfZ17/EjEDCQfRUwXMUt+zrfBup7Fz88/V3AANyDgIHsq4ChLAKGImgQkn3du0Fo/CD7uvf4EQgYyL4KGK7iln2NgOH7F+dlcj0CBuAeBAxkXwUMZREwFEGDkOzr3g1C4wfZ173Hj0DAQPa1YMAQ88czDfKByWH8OV47ta55e6f4xOIPLOZ88ed4LcjXXruva3I9AgbgHgQMZF8FDGURMBRBg3A/v/Brf9O6Bq72R/+4T99c515+6Td997ttx35vzcN67t0gNH6Qfd17/AgEDGRfiwUM0ZjP5U75jYtrLgUM64cwz87TBAzAMyBgIPsqYCiLgKEIGoT7+dN/3q9e18Cr3FrnXs77+pM+8sWb87CeezcIjR9kX/cePwIBA9nXQgFDPDA5l4lnGsQtiJKYNl99EH+fORcwzKFFhAkfWgximbjFUU4LBQzAMyBgIPsqYCiLgKEIGoT7+fFv+b6Xxv2Wy6F/8dN+4mdvTn/fVxHMVzB87Bu+Y3Me1nPvBqHxg+zr3uNHIGAg+/oEAcMpf2hxJm+JFE3/LSIYyHkibJg5FzBcWq8rGIBnQ8BA9lXAUBYBQxE0CN/G5dC/6GoB7uneDULjB9nXvcePQMBQw1/yG3/+y79Zfvyn/9gf/m1/6NdvznOv72MbfL8WCRjiaoV8fX11wkzcHinnyysRglMBw7XrzXkEDMAzIGAg+ypgKIuAoQgahG/jcuhfFDBwT/duEBo/3q+e68L36d7jRyBgOG023MNf+Cu/anOeU/7cr/+yd8s+olkf28/1/eO/+5d/YFqsP0KBmPZFX/EFH5j2Gs9tgzV9goDhmqb9HByc49R659dn5tfnQGJNznPNvp4j1yNgAO5BwED2VcBQFgFDETQI38bl0L8oYOCe7t0gNH68X+P2acthv9mtde7lvK/GuZruPX4EAobTRuN+OUQvfuZnf2hznlNmw/+1y53yXPM//p7TwnnaaxQw9LNIwDA/J+Ecp9Y7vz5z7Xpznmv29Ry5HgEDcA8CBrKvAoayCBiKoEH4Ni6H/sVLjbf41XHM9+W/9ve8/D2advlaPL8hXw/jV8Kxvg997kferT+N1889WyGeF5Hzxi+P19Pzl9O5v1/znX/8ZV9iH3K52K7nNzyXezcIjR/vV8914ft07/EjEDCc98Mf+ZH/x/7W3/fNm/Os/XW/45e8W+a1Vz6c8torGL78F3zRB6a9xr0Dhti3WHfs69Z0Pt6DBwzzg6NdwQBUQcBA9lXAUBYBQxE0CN/G5dC/eClgyPkiSPhpX/KJd39Po3EY88X09bQtT21vXj7XORuv5fIxb4YcW8Z+rpfn27h3g9D48Twup+PFS2MKea17jx+BgOG8822S4rZHW/OszUZ6+KhnGbyPqwv23sZHv+jz3q1/azofb5GA4dr5T91KaV5+Zn7dMxiAKggYyL4KGMoiYCiCBuHbuBz6F68NGOarEqKBHw3/+YqB+GVzNP0/52d+5cu0CAHS+OXyfKXB1q+Jrw0Y5mAh9j1+pRzLxvR5Wry2Xgffv3s3CI0fz+NyOt7V5dZ08rXuPX4EAobz3nKbpLyaIK5+2Jp+iwIG3mKRgCGuLoiHPsf83xsvbBDz/ODi1jynAobg0nq/azGXFTAAz4CAgeyrgKEsAoYiaBC+jcuhf/HagCGMBn4ECVvzXfLn/OY/cnab1wYM6dbtViK4yOkRdKyn8/27d4PQ+PE8LqfjRQEDH+Xe40cgYLjs3Bi/dJuk+fZIcfXD1jy3KGDgLRYJGIL5NkkRBnx0MYmrD35gMaZFYPDhxZlzAcMcIMQ28lZJsUyuMxUwAM+AgIHsq4ChLAKGImgQvo3LoX/xNQHDreFCGtuK9cSVD+tprwkYzt3LPa9iiCsmtqbz/bp3g9D48Twup+PFc2PK/KyV+Hs8S2W+9VqMDfFaTvv8j3/Dy/rmK6DCvFrq3JVK87a2nusSy8e0HG8iBI19ma+EurQN7uve40cgYLjsr/gnv+5dTVy6TdKp2yN9z7/ybS/LRpM9roTIecK44uGLvuILzjb1LzX/86qJc/sX+xDrmZ8rEX+O9xfTL20jApPY/3n5NF7P9czOV4Cccyu4if3Nbc7zntoWP9VCAUMwhwFbRriwdaujcwFDBArrIGH22xe/f/z5NfsaIcd6Xae8ar3jFLxjvAwcDwED2VcBQ1kEDEXQIHwbl0P/4rUBw6X5rjEDhnA97TUBw7mG37lt8P27d4PQ+PE8LqfjxXNjxVznESDMt15Ls77nmj/n1ngRXhpTcqyIafkQ+VN6SPTbuPf4EQgYLhuN7uVQvXjpNknZ6I8m+Pz63Lw/Z8w3L7e1/FbzP6ett5tGAz/3bctYLsKJ/Pt6G/H3ef5Trrcf292ab+16e7HcOohZG0FHnJt5OX7QNwoY4uqDPE/rqw0u8YnFuIIhb20URkAQQcCpdeX2YpktImSIqwryFkthbCMb//HneO01AcP8Hi8pYABeg4CB7KuAoSwChiJoEL6Ny6F/8VJwcO18afwKOH91PP8SeO16OQFDT/duEBo/nsfldLx4bqyY6zzDhRgnYswI40qFGENi3nx2S1xVEH+OZdOYdx5fcplT2zoXMMzriW3lNuYrK2KevLKC78+9x49AwHCdcYXBcrhePHWbpPn2SOtf2Mcv8aNhHlc4xJ+joZ5GY39u/m89GPqegCGa8HOzPuaJfY31xL5sBQ/rbWRAEcch9iX3Pdcxr3/93nO++cqHeflwnj/2N/cp/hvryyAhjs18LGKd87L8oG8UMOAGxil4x3gZByNStEzXIlk7JAIGsq8ChrIIGIqgQfg2Lof+xUvBwbXzhdH0y/kvuV72UjNQwFDTvRuExo/ncTkdL14bMITzLZFe6/zMla0x49KYMo8VESBsjStzyHDu1mzcx73Hj0DAcJ3R5F4O14unbkM03x7ptb+sn9e/dRXDPQFDBAA5PQKC9fTY11gu5wm3tnHO+VZIW/sQztvYmp7Ox/FUmDO/pwhLtuahgKES4xS8Y7yMgxGXu7wb3OKFIyJgIPsqYCiLgKEIGoRv43LoX3xUwDDfYiSahtH8WzcOzzX/BQw93btBaPx4HpfT8eK5sWKu87g64d6rAnJdEW6up70mYDj1fJn5OQ4RNmzNw/3ce/wIBAzXm7+qP3WbpJy+1cS/xlg23Aow7gkY5isHtq6OSOcA4LUBQ5jLn7qq4JqAIcKOnCeChq150mvnO7IChjqMU/CO8XIZ4uEY8RCN+T5UYTzgIu5v9Zr7Tx2ZZw0Y4j5jW/cvi/Md5/2h51fAQPZVwFAWAUMRNAjfxuXQv3gpOLh2vnwI67lbiZxr/gsYerp3g9D48Twup+PFc2PFpTp/ree2+ZqAYT1tNuc59764j3uPH4GA4XrnX9avG/Dnbo90rbn8VkhwT8BwbtrspW1c8lKAcE3AMB/HS1cm5Pouva8jK2CowzgF7xgvPz3ReM6nZF8yggac5xkDhniAyRwqnDLCh/g83I2AgeyrgKEsAoYiaBC+jcuhf/FS0+yR851r6AkYerp3g9D48Twup+PFc2PALQFDLBNXD8y1vXZrmwKG+u49fgQChuudm9/rX81n+BBXMcyvr43Gfcw7N9vXbjXMbw0Y5ocsn7q1U3pNwBBXQMR6Yjt5xcaWW8vO73lrejjvw7VuHS9+UgFDHcYpeMd4+amJZnI8J2AuyGgyR0M6XF/REEEEzvNsAUOcx3l/wjiP4fpqlTDO/90IGMi+ChjKImAoggbh27gc+hcvNc1eO19cybA1fb5ferieLmDo6d4NQuPH87icjhfPjRWvCRjiSqi5ns+5tU0BQ333Hj8CAcPrzKb6OkjI10/drmfrOQen3GqY3xowxLw5LdYxT1t7aRsRLOT0S66XDfcKGE4dcwoYKjFOwTvGy0/N3HyOX7h/dHGLaJpHQzoCB5zn2QKGvDolzm/cIml9hUK8tr664e7bJQkYyL4KGMoiYCiCBuHbuBz6Fy81za6db34Yavz55/zmP/Lyevx3npaulxcw9HTvBqHx43lcTseL58aK1wQMn/Mzv/LdvPHnL/2m7/6UeXL61jYFDPXde/wIBAyvc26y5y18rrmtTzyXIeeJP2/Nl9P3ChjuuYJhfqhyPGMhbgO1fpD1pQDhtQHD1vvk6xQw1GGcgneMl5+a+Rfs0WjG/TxbwBC3tbp066M49/M+3x0kCRjIvgoYyiJgKIIG4du4HPoXLzXNrp0vgoR4/kLOvzamzQ3D9fIChp7u3SA0fjyPy+l48dxYcW3AEOPJNes7N4+Aob57jx+BgOF1zrccyl/OX7o9UtxWKJfZCg/Sc/PcGjBcmjZ7bhvxYOt4Pd7jOlhILwUIl6aHc1gTocbWPLxeAUMdxil4x3j5qXlXrIt3/2odLzxbwHAt81UMd98KS8BA9lXAUBYBQxE0CN/GDAPi6oKt6WnO9/kf/4bN6bNxS5OYLx/4HH7ocz/y0tyLaV/4tb/p5bVY53rZj3/L971bZuuXyvOyeXXElhlibG2D79+9G4TGj+dxOR0vPiJguHa+nEfA0NO9x49AwPB652Z7/P3S7ZHmqwiiib81T5jzPDpgyP0NT4UD8XpcmZDzrbeRr58LKR4RMFwbxvA6Xxsw4HlYauDpeVesi/cGDHF7pbjlUjSn17fcide+cfHSA4TzeRDzcwBiv+Lvua5Yd/zCfmt/45f487xxhUbMe267sd85f+xjktud30v8/eOL53htwPDhxdjH+VkYsc04Zu/zqpK8lVIoYCB5UgFDWQQMRdAgJPu6d4PQ+PE8LqfjxUcHDKdC0PnWawKGnu49fgQChtc73yYpr14I4+qGrfnngOFUCDGvZ6uxfk/AsN7f9fR1uBCeChgirJhfT+OWSfPyW/PM+3HqWIVzEBHr3ZonPbceChgqs3z+n5518/xW5ub0OWN7p57zEOR8sb4IBaLxPi+/NhvwMe+5fTi33TkQiIAk1jWHFFueu4XQawKGuH3RPO+WETycO2aPQsBA8ioFDGURMBRBg5Ds694NQuPH87icjhcfETCE85VQcQVTXAUVr8c65nAgFDD0dO/xIxAwvN75V/bpqcZ7Ol9FELf+ySsJopE/N9TDRwcMsb95lUWYz4CI9cR6c9o8z3obcwASf4515rrnaem8bDqHEBFo5D7M6wvn21CF8Z5i2Zg3jOUirMhQZL2v/BEFDHVZPttPz7qBf65xfop1Qz0a4tE4j2Z9uG7Wx1UFp5jXMf+iPxresa7YvzkUCeOKgnne2F7OO893arvz/sd+z+uKP+f7mJ9XkfNucW3AsN6/eVvxHub3Gdu+dPXHvcz7cuq9XY2AgeyrgKEsAoYiaBCSfd27QWj8eB6X0/HiowKGuFVazrtl3H4tjD8LGHq69/gRCBhuc/2L/0sPUJ6fLbBlrC/X+eiAIYwG/RwgrI0m/7yP622sQ4q1MW1+kPW8bBqhyhy0zK63F/tybnuzrmI4rYChLstn++mJ2/OsG/bRzH7NrXlyHdGUjj9vMd+GKDx1m6F5njDWu5431rXe55x3/Uv/9Xa33tc6EAhjXfH6mnUosPV+rwkYYj9y+qltxbrnsOOW8Oda4hjndsJLt4G6iICB7KuAoSwChiJoEJJ93btBaPx4Hq95rsulZ62sjfljfbnuMBr9H/uG73iZns9cif9uLXtuW7lshBTrabO57a1tcF/3Hj8CAcNtzr/Gj0b4/Av8U0YjPBr5c+M8f50f07NBH/9dLxtXPZzbVq7zXNCRVxvM249tZXN/vnJgq2kfAUGsfw4JIhSJ8COmzfu4XjZd70P8N/Yhr+iYzXXGMcrthbH9eC22e81xP7IChrosn/USnGrYZ9DwqF/Oz835+JX+FvP2twKDJMKMa+edt7vVpF8HAufWFcxXMmz90v+agGFex7lmfoQM87r2uophvj1S7NvdCBjIvgoYyiJgKIIGIdnXvRuExg+yr3uPH4GAgeyrgKEuY4guQTSyTz3DIBru88OPbyVChVznqec9zNs9t811E/9cIDBfLbD1bIHXrCuYw42tZvylgGG+WuCaZv58i6m7ryzYYL2/r7l65SQCBrKvAoayCBiKoEFI9nXvBqHxg+zr3uNHIGAg+ypgqMsYoksRzeZTQUPcrueeX9DPAcNWoz+Ytxf7corYj3nec8wN9GsChktcmv/S9Pk4XPOsg3n+U1d+3Eocx/lqilPn5dUIGMi+ChjKImAoggYh2de9G4TGD7Kve48fgYCB7KuAoS5jiC5JXNGwft5AeE3IENOjyR7N8Nk5uLg3YAjmec8xN/wfETCsb1u0vuLh0vrWx2F9nNbO88ffH8l8dURcqXJPgPQBBAxkXwUMZREwFEGDkOzr3g1C4wfZ173Hj0DAQPZVwFCXMUSXJprn84OGw1NN7rh9z3reU1YOGIJ5/vV+viZgeK2PDBjWAdKlW0O9CgED2VcBQ1kEDEXQICT7uneD0PhB9nXv8SMQMJB9FTDUZQzR5VnfRmfruQFbVzucU8Bwm496BsP8XIrwIc9dmBEwkH0VMJRFwFAEDUKyr3s3CI0fZF/3Hj8CAQPZVwFDXcYQ3YJ1Q3omHsY8T4vb7kQjfH27nfj1fc5TOWBY3yIp/j7zmoDh0bc8uobdw4VAwED2VcBQFgFDETQIyb7u3SA0fpB93Xv8CAQMZF8FDHUZQ3QLzjXN56sbzjXMuwQMEZ6cm//S+uLBzjntmoc8P5L3Ei4EAgayrwKGsggYiqBBSPZ17wah8YPs697jRyBgIPsqYKjLGKJbcO4Khvn19a/5Z6oEDJeeRTDfDuqWwGI+lvHMivfFewsXAgED2VcBQ1kEDEXQICT7uneD0PhB9nXv8SMQMJB9FTDUZQzRT0s0yqNBfi4UCOJWR/PDm+MWSDP5eniuOR/L5XzPHDDEe13f3imJ139oMeeN20OtuRQwxDrm6Zfe5yN4r+FCIGAg+ypgKIuAoQgahGRf924QGj/Ivu49fgQCBrKvAoa6jCH6aYlmezYY4lY9W0FDNL/ncCFcN8TnWyRFaLEm1jtvK3zmgCGM97QOS9ZBSwQNW0HEen1bzLdJivVcenhzTL81FHh0uDDv+8n9FjCQfRUwlEXAUAQNQrKvezcIjR9kX/cePwIBA9lXAUNdxhD9tMy3LEqjsR5N+HAODtKt5vS6gR3LxbrD+aqF2WcOGOYrFGL/431EU31+PTzVXL8mYFiHFWEct9hOHrv4c+xvTo/t38K83/HnPL+XjH3YItcVrq9meYeAgeyrgKEsAoYiaBCSfd27QWj8IPu69/gRCBjIvgoY6jKG6Kclmtzzr9HPGY3pc798n59LsGUsPz8cORrYW8zLvFXAEFcurJv/a88di2sChiCOf+zPPO85Tzbzz7Del9d4KtCY54nPzyYCBrKvAoayCBiKoEFI9nXvBqHxg+zr3uNHIGAg+ypgqMsYop+euIVRPEsgmt1bv3aPaVu3AloTAUI0wed1RKM+fgmfy+e0rVspBdnYj/kuPRsi1xW//D9HBAY571aTfisQiP2N/Z6DhthO7Pel/Yrpub1Y/hKx/VjvOtSIdcTxjyb+yVsRXUHuy2s9FWjM82w9g+IFAQPZVwFDWQQMRdAgJPu6d4PQ+EH2de/xIxAwkH0VMNRlDNF4cq694gCfDE/yOEVwcTJ4EjCQfRUwlEXAUAQNQrKvezcIjR9kX/cePwIBA9lXAUNdxhCNJ0fAcD3zrbBOPaPhBQED2VcBQ1kEDEXQICT7uneD0PhB9nXv8SMQMJB9FTDUZQzReHIEDNcRVytcfesnAQPZVwFDWQQMRdAgJPu6d4PQ+EH2de/xIxAwkH0VMNRlDNF4cgQM1xFXLOQximN2FgED2VcBQ1kEDEXQICT7uneD0PhB9nXv8SMQMJB9FTDUZQzReHIEDJeZr16Ih05fRMBA9lXAUBYBQxE0CMm+7t0gNH6Qfd17/AgEDGRfBQx1GUM0npx4cPHVt/45KB9fzONz8sHOMwIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqgQUj2de8GofGD7Ove40cgYCD7KmCoyxiigeMhYCD7KmAoi4ChCBqEZF/3bhAaP8i+7j1+BAIGsq8ChrqMIRo4HgIGsq8ChrIIGIqQDQSSfR3l/nBy/VvNSZK13Xv8CAQMZF8FDHUZQzRwPAQMZF8FDGURMBQhGwgk+zrK/eHk+reakyRru/f4EQgYyL4KGOoyhmjgeAgYyL4KGMoiYCiCBiHZ170bhMYPsq97jx+BgIHsq4ChLmOIBo6HgIHsq4ChLAKGImgQkn3du0GYzQOSfR3lvgu5ja3mJMnaZn2jHmOIBo6HgIHsq4ChLAKGIggYyL4KGEje6yj3XcgximRfUY8xRAPHQ8BA9lXAUBYBQxHyH/9bzUmStc36HuX+cLIBufXLRZK1FTCQfISoxxiigeMhYCD7KmAoi4ChCPmP/63mJMnaZn2Pcn84Agayr+8zYNgav0jWNusb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaOB4CBrKvAoayCBiK4Ms92des71HuD0fAQPZVwEDyHrO+UY8xRAPHQ8BA9lXAUBYBQxF8uSf7mvU9yv3hCBjIvgoYSN5j1jfqMYZo4HgIGMi+ChjKImAogi/3ZF+zvke5PxwBA9lXAQPJe8z6Rj3GEA0cDwED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxiigeMhYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHrG/UYQzRwPAQMZF8FDGURMBTBl3uyr1nfo9wfjoCB7KuAgeQ9Zn2jHmOIBo6HgIHsq4ChLAKGIvhyT/Y163uU+8MRMJB9FTCQvMesb9RjDNHA8RAwkH0VMJRFwFAEX+7JvmZ9j3J/OAIGsq8CBpL3mPWNeowhGjgeAgayrwKGsggYiuDLPdnXrO9R7g9HwED2VcBA8h6zvlGPMUQDx0PAQPZVwFAWAUMRfLkn+5r1Pcr94QgYyL4KGEjeY9Y36jGGaOB4CBjIvgoYyiJgKIIv92Rfs75HuT8cAQPZVwEDyXvM+kY9xhANHA8BA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nfqMcYooHjIWAg+ypgKIuAoQi+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0cDwEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrG/UYwzRwPEQMJB9FTCURcBQBF/uyb5mfY9yfzgCBrKvAgaS95j1jXqMIRo4HgIGsq8ChrIIGIrgyz3Z16zvUe4PR8BA9lXAQPIes75RjzFEA8dDwED2VcBQFgFDEXy5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPpGPcYQDRwPAQPZVwFDWQQMRfDlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaOB4CBrKvAoayCBiK4Ms92des71HuD0fAQPZVwEDyHrO+UY8xRAPHQ8BA9lXAUBYBQxF8uSf7mvU9yv3hCBjIvgoYSN5j1jfqMYZo4HgIGMi+ChjKImAogi/3ZF+zvke5PxwBA9lXAQPJe8z6Rj3GEA0cDwED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxiigeMhYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHrG/UYQzRwPAQMZF8FDGURMBTBl3uyr1nfo9wfjoCB7KuAgeQ9Zn2jHmOIBo6HgIHsq4ChLAKGIvhyT/Y163uU+8MRMJB9FTCQvMesb9RjDNHA8RAwkH0VMJRFwFAEX+7JvmZ9j3J/OAIGsq8CBpL3mPWNeowhGjgeAgayrwKGsggYiuDLPdnXrO9R7g9HwED2VcBA8h6zvlGPMUQDx0PAQPZVwFAWAUMRfLkn+5r1Pcr94QgYyL4KGEjeY9Y36jGGaOB4CBjIvgoYyiJgKIIv92Rfs75HuT8cAQPZVwEDyXvM+kY9xhANHA8BA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nfqMcYooHjIWAg+ypgKIuAoQi+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0cDwEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrG/UYwzRwPEQMJB9FTCURcBQBF/uyb5mfY9yfzgCBrKvAgaS95j1jXqMIRo4HgIGsq8ChrIIGIrgyz3Z16zvUe4PR8BA9lXAQPIes75RjzFEA8dDwED2VcBQFgFDEXy5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPpGPcYQDRwPAQPZVwFDWQQMRfDlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaOB4CBrKvAoayCBiK4Ms92des71HuD0fAQPZVwEDyHrO+UY8xRAPHQ8BA9lXAUBYBQxF8uSf7mvU9yv3hCBjIvgoYSN5j1jfqMYZo4HgIGMi+ChjKImAogi/3ZF+zvke5PxwBA9lXAQPJe8z6Rj3GEA0cDwED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxiigeMhYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHrG/UYQzRwPAQMZF8FDGURMBTBl3uyr1nfo9wfjoCB7KuAgeQ9Zn2jHmOIBo6HgIHsq4ChLAKGIvhyT/Y163uU+8MRMJB9FTCQvMesb9RjDNHA8RAwkH0VMJRFwFAEX+7JvmZ9j3J/OAIGsq8CBpL3mPWNeowhGjgeAgayrwKGsggYiuDLPdnXrO9R7g9HwED2VcBA8h6zvlGPMUTjSfj+xTgpP7T4iXgB+yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNJ6Ajy5mMyWMsAE7ImAg+ypgKIuAoQi+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0noCPLWYzJRQw7IyAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0XgS3CLpPSJgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaOB4CBrKvAoayCBiK4Ms92des71HuD0fAQPZVwEDyHrO+UY8xRAPHQ8BA9lXAUBYBQxF8uSf7mvU9yv3hCBjIvgoYSN5j1jfqMYZo4HgIGMi+ChjKImAogi/3ZF+zvke5PxwBA9lXAQPJe8z6Rj3GEA0cDwED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxiigeMhYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHrG/UYQzRwPAQMZF8FDGURMBTBl3uyr1nfo9wfjoCB7KuAgeQ9Zn2jHmOIBo6HgIHsq4ChLAKGIvhyT/Y163uU+8MRMJB9FTCQvMesb9RjDNHA8RAwkH0VMJRFwFAEX+7JvmZ9j3J/OAIGsq8CBpL3mPV9C2OIAID3i4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrO9bGEMEALxfBAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfd/CGCIA4P0iYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHr+xbGEAEA7xcBA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nftzCGCAB4vwgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPq+hTFEPAsfXYx9+qGXv72eWC6Wj/UAeGYEDGRfBQxlETAUwZf7/fzCr/1N6xq42h/94z59c517+aXf9N3vth37vTUP65n1vZzXXRAwkH0VMJC8x6zvWxhDxLPwscX8d/It5LKxHgDPjICB7KuAoSwChiL4cr+fP/3n/ep1DbzKrXXu5byvP+kjX7w5D+uZ9b2c110QMJB9FTCQvMes71sYQ8Qp8oqCSz7qioG3Chg+sfi9iz+4mOuIP3/74ocWATwaAQPZVwFDWQQMRfDlfj8//i3f99K433I59C9+2k/87M3p7/sqgvkKho99w3dszsN6Zn0v53UXBAxkXwUMJO8x6/sWxhBxirnhf87KAcMPLOZyW8Ztl9xyCXg0VQOGz/wb/vatgeIqY9mtde7lX/u3/AMv2/0rf8xf9cM/4+/9rs157vV9bON9+MX/4O97eS+f/pmf9+58xXuKc/Y3/Z3furkMTytgKIuAoQi+3L+Ny6F/0dUC3NOs75di3wEBQw1/yW/8+S/jzY//9B/7w7/tD/36zXnu9Yu+4gtetvHhj3z25nTWU8BA8h6zvm9hDBGnmBv+r70q4BbeImCIAOG7FudlPrwYVzTk+r5/EcAjqRowfOhz/5YcGF5tLLu1zr2c9/Wn/a3/0OY89/o+trG3ESDkezhlBA9by3JbAUNZBAxF8OX+bVwO/YsCBu5p1vdLse+AgOG02XAP//Hf/cs35zllNOljuc/87A9tTn+tv/BXftXJfYnAIYKHmBb7PE97jR/9oh/5Yc3WdNZTwFBbz4HiW5v1fQvLZ+EcRwgYzjHfMilCBwCPomrA8AU/9zteGulrf8pHf24OFi9N9615Ytmtde7lfHXB3/J1v3tznnt9H9vY0zgned7iPcT7iSsxwv/6l/1jH7iiYWv5R/hjf8JPfln/+77CZU8FDGURMBTBl/u3cTn0LwoYuKdZ3y/FvgMChtP+in/y697V+Zf/gi/anGfL3/r7vvmm5c55LmCIv+e0cJ72GgUM/RQw1DZut7gc4pvdWudezvvq30V9zPq+heWzcI6jBwy/ZvHR6wQQVA0YThkN6eVtvRhhwtY8fD7nAOHULZ7iCocIAbamPcLc/vu+wmVPBQxlETAUwZf7t3E59C9e+iIdvyKM+b781/6el7/Hl/B8LZ7fkK+H8au/WN+HPvcj79afxusfO/NshXheRM4bvyRcT89fQub+fs13/vGXfYl9yOViu+e2wfdv1vdyfnZBwHDa7/lXvu1dbcQVAlvzbPlzv/7L3i332isfTnntFQz3BBp7BwxxNUes+56rLPg6BQy19RwovrVZ37ewfBbOcU/AkA9OjlsQ5TrieQfx4ORTVwNcEzDEeufnJsSf47UgXxMwAM+OgIHPYJ6zt7wFUu6DgAFvzBcs/vnFd5/J4dcs4snw5f5tXA79i5cChpwvgoSf9iWfePf3NBoBMV9MX0/b8tT25uVznbPxWi4f82bIsWXs53p5vo1Z38t52QUBw3nn2yT9ut/xSzbnWZuN9EfdHik8FzA8yr0Dhlx3bGdrOh+vgKGvy6F/8dK/Qch7zPq+hZcB4jS3BAwfWpyfX7BlhA5b65u3t0U8K2Fez+w87VFhwPw+ADwSAQOfwTxnb9ncf4Z9eLQChlL8hMXvWPzLi+8+j8N47dMW8WT4cv82Lof+xWsDhvmqhGjgR8N/vmIgfqkYTf/P+Zlf+TItQoA0fok4X2mw9evAawOGOViIfY9fHcayMX2eFq+t18H3b9b3ck52QcBw3tfeJmm+PVJcybA1zy0KGHiLAoa+Lof+RQED9zTr+xZeBojT3BIwxBUKuUxcAZBXK0TwEFca5BUN8d/1lQznAob5aoIIE2J9QSwTD2HOaeEjAoZYf64vggYAj+SoAUPc1z/myWZyPLcg/pzLxrMcct6YFn+P6fF8gJwnjFv2xLQv/gd/37v51+a2Ytmt+XKdeWug+G88ByC3EdOfYRtp3KpoPlanjH3aWn7Leblr9uGU8b7ifeTzFMJz5+iaB0vHsVkvV0UBQwl+6uJ3Lv6lxU/5/A3/wCKeEF/u38bl0L94bcAQRgM/goSt+S75c37zHzm7zWsDhnTr9gkRXOT0CDrW0/n+zfpezskuCBjOG7dJytsPXXObpPn2SBE2bM1ziwIG3qKAoa/LoX9RwMA9zfq+hZcB4jRzw/+UERQkERjk6xEIbPHRxZwnwoiZcwFDPnA5woQtHn0Fw3z1QuwzgEdy1IAhpsU80SiPACEb8LM57zXN9Fj+1MOjc1thNvhnc1osH43w/Pva2MapBzi/j238t/7hP/SBZyVc8tzxXzvvU2zj1D6c868dD7o+ZwQK8zIZzFxyXqaSAoan5acs/srFP7n4KZ+3lX9q0dULT4ov92/jcuhffE3AcGu4kMa2Yj1x5cN62msChnP3Zs6rGOKKia3pfL9mfS/nZBcEDJeNKxeWQ/Xipdsknbo9UoQNsZ5ormdgkca8MS2epTAvM3spYMh1nrtqItYf28l9DOMWULm+SwHDL/mNP/9lng9/5Eeupkrj9bjaY73MfAXIKU8FNxHu5Dbn+U9ti5+qgKGvy6F/8dK/QfL/6XlV4ny1oudA8ZJZ37ewnNNzvDZg+MbFfD2vMNgim/fx/ISZUwHDHEp8PF44Qc5zb8AQV1rkuk4FJQDu4egBQ/y6PcOF+HO8Hs3u+VkA8Uv/+Hs0sKNBHdsI1836WH7eRnpt8z/3I/4by+Q25oAj/rxePnwf25jfaywbgUO8HlcGzFdDRNM+1pvTrzHWkfuWxvauvZphfv+xL3NAEe9vDkbi7zkt9jH2NczpMW++Ft4SdjyLAoanIQKCn70YVyr86cV3n7cLRgDxWYt4Uny5fxuXQ//ipS/31853jbGOXN962msChrmhsPbcNvj+zfpezscuCBguG6HCcqg+WTtnbpM03x4pAoF52rpJvmU02k8FGJcChpwW21lPCy81+iOYmPdxvXxsc57/lOvtR0CwNd/aeZkwjuUchGwZQUeEEOtl+SMKGPq6HPoXr/03SPx/33Og+Fqzvm9hOZfnmBv+1zTtMzg4dZVBMt/uaOZUwDC/fi64yHnuCRgizMjbOF16HwBu5egBQxrN7K35rnFuvMf219PnbW1Nz2lhNLe3murzLX+2Gt57byMa8Tnt1LHK5aPBvzX9krHNeR/m7Z1r8sd7yXlPbTv2P9cd739rnlzHqYClogKGNyNue/S1i799MUKCrecqnDNul/TNiz9qEU+ML/dv43LoX7z2y/2l+dK4FdLnf/wbXuY/9wV8vdy1X+7DmHc9PY3t5nxb0/l+zfpezscuCBiuM68QOPVr+3C+PdL6aoS4UiAa4jFPNPujYR9GoDBfIbG+8iG9J2CYg48wtpfbn/d5dmsd8d7jfcS+5PJhhAhzGDBfXRABQM6X0+M4zMvHuudtrW9LFevLICGO63wsYl3zsvygAoa+Lof+xWv/DeI5ULzFrO9bWM7jOV4bMOSzEB4dMJyaf03Oc2vAMIcLcXXFuTADwD0IGG5viKfzNudfx6fXNv+jAX7qV//zrXzWt/kJ997G/B631h/GFR45z9b0a4x9i/eyvpohjKBha9/n7Z674mE+Rlvz5TQBA15BXJnwMxa/fjHChH9t8S8svvs8vdJYNh707KqFIvhy/zYuh/7Fa7/cX5ovjGAh57/ketlrv9yH5764Cxiey6zv5XzsgoDhOucQ4NRVBtlkv6XpPa8/mu7r6fcEDBEK5PQIA9bTt64WWM9zyWj857KnrqK4ND2dj8U6fEjnKyMu3bbqyAoY+roc+hev/TdIGA18z4Hia8z6voXlPJ7jWQKGuC1Svr7XFQzCBeB9ImA43TC/1kvbvLStnHausf3W25inba0/nK/k2Jr+Gk8FDXH1wTpkiNdy2vz62kvvIacJGLAiriKIKxLiFke/bDGChD+8+OcW331u7jSCiVi3Zy0Uw5f7t3E59C8+KmDI+xOH+avCuEfxPM+55r+AoadZ38v52AUBw3XOVwFEA/zc9K0m/iUjNMjltxrmtwYM8cv/nHbq6ohwvmog3JrnkrHtWPZUwJLrPhcwzPu7dZxnr53vyAoY+roc+hev/TdI6DlQfK1Z37ewnMdzvDZguPZKg1O3UjoVMMyv7/EMhggThAvA+0TAcH3AEL/qj6sdsqG95dY2L20rpz1zwDDfhiiuGJinhdH0z1sQPbJBn0FDbjtc36Jpnnattx6jagoYLhIN/c9d/JLFePBTNPl/y+LvXfwTi48MEWbjlkmx/t+wGA98RlF8uX8bl0P/4rVf7i/Nl7cfiC/X62AhPdf8FzD0NOt7OR+7IGC43vyVfzTi85Y96bnbI13jHDBEmLCefmvAcGm9sxkQhFvTL3lp+Zx2LmCIcCXnu3RlQm7v3PqOroChr8uhf/FR/wa5Rv8GOZ5Z37ewnMNzvDZgmOePBz5vMT+weT3PvPyaDAAinNjiuxZz2dcEDBEmRKgQy/3g+DuAvREwXA4YTj0bYMutbV7aVk575oAhnK9QiNsp5ZUEET5E8JLTtm7hdK+xzlx/OE+bX7/GuCpi65kOOV3AUJZo1EdYELcsisDgqxZ/0eKvW4zQ4Hct/sHFuFrgzy7+xcV35/09+KcWv3vxqxddqdAEX+7fxuXQv/ioL/fXzOfL/fHM+l7Oxy4IGK53DhHm5wyEGT7E7Yjm19fGcvk8hlzX2kcGDK+5ldClgCCM8CSOQ8w7X/GwdmvZnHYuEJjf57UKGE4rYOjrcuhffNS/QVLPgeJs1vctLOfwHK8NGIK8TVIYVzR8eDGIxn38SG++UmDNuYBhDhBivRkExDIZEKTX7uscLsR+RfgB4H0gYDgfMMy/zA/j1/vr5vQzNP/fxzbmZx2ccn11wSOdrxyZ32O+dm8w8Kj1PJM3BAz5a/69jOZ6NP1PGVcQfOvKb1uMcCD8o4t/bDEenBwhwZ9ffHfunsh4lkLcRin2P26t9BMW0RBf7t/G5dC/+Kgv9znfqdsCzPcmDtfTfbnvadb3cj52QcBwvfNtkOYgYX59HTzM86yfc3DKRwYMl5abvRQwnHog9JZby+e0RwcMbpF0WgFDX5dD/+Kj/g0Seg4U12Z938JyDs9xS8AwN+1PGdO3mvnnAoZL6/32xQw3rt3XCDzW6zllBBAAHoWA4XzAMP9y/tT6Lm3z0rZy2jMHDPMtkvJWUfn3uCIg/r613Uca+57bnLeVwUPsxzz/a811HzxgiIb4u2PBq/wzi39gMY5dBChueXQgfLl/G5dD/+Kjvtz/tC/5xLt548/xK8J4Pf47T0vXy/ty39Os7+V87IKA4XXOVx7kbZLmxvv61klhvDaHCzF/BA7zPJduZfSIgOGeKxjmKyHiGESQsn6vlwKKnHZtwHApEOFlBQx9XQ79i4/6N4jnQHHLrO9bWM7hOebbGeWVCNcQYUDc/mi+miGMv8frefXBmtzeqWZ+LBdXL8StjHKdccukDBTy2Q7XBgxzoHFJAQPwSAQM5wOGa+a7tM1L68hpzxww5Prfsvk+X0kyvz7fumnrvV9rrkPA8MnjwA8YV0vElRO/czFuuRRBwucv/phFHBhf7t/G5dC/+Kgv9xEknLsdQUz7nJ/5le/+vl7el/ueZn0v52MXBAyvc26059UKl26PFPPlMlvhQbhXwHBpvbPnAoJzz59IHxEwzM9guOVh2fygAoa+Lof+xUf9G8RzoLhl1vctLOcQAN4/AobrA4atZwvELZTmW/dsbfPStnJahYAhrhI4d7xuMdYd+/UFP/c7NqfHMZ5DhLhaYp4+73eci3w2xJYxLa7G2Jr2qCshnkkBw13GMxrimQ3fvPg1i/FsB7c6wgfw5f5tzDAgri7Ymp7mfHHrga3ps/GlPubLL/ph/JIwvpjHtPyFYaxzvezHv+X73i3zpd/03Z8yfV42r47YMkOMrW3w/Zv1vZyTXRAwvM54BsFy2F6MQOGa2yNd86v8vQKGeX/jyoN52my8j/mZCuvp+fq5cOARAcO8v+fm43UKGPq6HPoXHxUwXDOfgOF4Zn3fwnIOAeD9I2A4HzDMtwaKxvPcBI/AYf3w561tXtpWTnvmgGGetjaOS6w3jOc0nGrgn3Le91hXBAjxWhjBQrw2T99a//qWTbEfsc9pPJQ65sl9XS8frkOMWC7O957PldhbAcMuxnMf4kHR8cDmCB/iaoafuogD4ss92des71HuD0fA8Hrn2yTF/f/jv9Gc35o3nMOBrRAirgiY1/nIgCGc1721/XW4EK7nydfjSob1tHC+SmNr+TD349yxCueg4lRok65vNcUPKmDo63LoX3x0wOA5UJzN+r6F5RwCwPtHwHA+YAijWZ3zbjk3pre2eWlbOe2ZA4YwmvQ5/ZJbV3ucMpr4W+tYG2HO+gHbaVyZMIcM54z5ttYR696aP9yav4I3BAzxoOW4FdAe/onFuNXQln9p8VOOezHjiod4SFUED/Gw6p+16BZKzfHlnuxr1vco94cjYHi962Z6eO5Bw/Ov8qO5Pj8LIda1fvjzowOG+bZDYTwDItYRZkASnruCYZ4v/hzvKV6P/87T0vXy4TxfXP0R2499i9fn+earQsJ4T3Gccp9jmXgPGVjEa/Py/BEFDH1dDv2LjwoYPAeKW2Z938JyDgHg/dMtYJibxOea3NksP/WL+LXRBF//Sj5/5R7T4+/xeqx3veylbeWyEWSsp6Xz+3qLbcQ6cx1xK6F437MRSMxBS3jNcU1j3thmBCC5nTD+HMc5zuW5Wx+lsS+xH+srS2K98fql4COWn89zrOfcMXt2bwgYno1PW/zcYTyw6UsWv2oxgpDwNyzGVRffsfi7hn90MQKNaPa/dXgR245g5TsX4zZLn7WIRvhyT/Y163uU+8MRMLzeuOJgOXQfcA4NtpwfBL3l3Hx/dMAQXtp+PO9g3sZ6+QgS1lc5zMa0CA3y7+vlw3VwMLueN47nue3NuorhtAKGvi6H/sVHBQyeA8Uts75vYTmHAPD+6RYwcB+jQb98XF4a7uca/Rl0hK+5ioH72CBgeBRxJUGEFPEchS9bjHAiHtYcjf/fvxi3PIow4t3nd0f/5OJvWfzZi65wKI4v92Rfs75HuT8cAcNtzoFA/JJ+a5610TSfm/DZlM/AIBvqWw83zodLxzx59cBsLhtBwnpaGttZbz/eR65v3sZ62TCClVj/fMVFvPcIJmLapeXD9T7Euk7tc65zvmVSLhOvxXa3jgV/RAFDXzMM8Bwo7mnW9y0s5xEA3j8CBl4yAoXlo/Li1q2TZuMKgJx360oLvl8FDDcRVxnE7Y0+sRghxG9fjIc9/6nFv7z47jP+AOO2Sr9zMUKPH7WIYvhyT/Y163uU+8MRMJB9FTCQvMes71sYQwQAvF8EDLxk3L5o+ai8eOl2QfNtkk49L4HvTwHDLnzeYtymKR7wHM9biFsgRVDw7rN/o/Hw6FjnT1hEEXy5J/ua9T3K/eEIGMi+ChhI3mPW9y2MIQIA3i8CBl7j/EyDCBnW4UE++yDnOfUgZb5fBQzvlZ+6GM9Z+LbFuOLhLyy+q4lXGMvFcyXcPqkAvtyTfc36HuX+cAQMZF8FDCTvMev7FsYQAQDvFwEDrzEChfnhy+eMcOGaBzJzfwUMb06EDl+7GFc6/OnFT6mXM8YzIeKh1nhifLkn+5r1Pcr94QgYyL4KGEjeY9Y36jGGaOB4CBh4rREaxHMV4oHP67Dh0z/z816uYIgrGbaW5dsoYHg64kHTX78YD5a+9rZKv3QRT4ov92Rfs75HuT8cAQPZVwEDyXvM+kY9xhANHA8BA9lXAcNTE7dA+urF37P4lxbjPG0ZD5aOWy/hCfHlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCjDZy3+lsVTz234c4t4Qny5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMrxUxb/8GKcs7Wft4gnw5d7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSQ/anErZPiVi3gyfLkn+5r1Pcr94QgYyL4KGEjeY9Y36jGGaOB4CBjIvgoYyvKti3HeZuM1PBm+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0cDwEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrG/UYwzRwPEQMJB9FTCURcBQBF/uyb5mfY9yfzgCBrKvAgaS95j1jXqMIRo4HgIGsq8ChrIIGIrgyz3Z16zvUe4PR8BA9lXAQPIes75RjzFEA8dDwED2VcBQFgFDEXy5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPpGPcYQDRwPAQPZVwFDWQQMRfDlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaOB4CBrKvAoayCBiK4Ms92des71HuD0fAQPZVwEDyHrO+UY8xRAPHQ8BA9lXAUBYBQxF8uSf7mvU9yv3hCBjIvgoYSN5j1jfqMYZo4HgIGMi+ChjKImAogi/3ZF+zvke5PxwBA9lXAQPJe8z6Rj3GEA0cDwED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxiigeMhYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHrG/UYQzRwPAQMZF8FDGURMBTBl3uyr1nfo9wfjoCB7KuAgeQ9Zn2jHmOIBo6HgIHsq4ChLAKGIvhyT/Y163uU+8MRMJB9FTCQvMesb9RjDNHA8RAwkH0VMJRFwFAEX+7JvmZ9j3J/OAIGsq8CBpL3mPWNeowhGjgeAgayrwKGsggYiuDLPdnXrO9R7g9HwED2VcBA8h6zvlGPMUQDx0PAQPZVwFAWAUMRfLkn+5r1Pcr94QgYyL4KGEjeY9Y36jGGaOB4CBjIvgoYyiJgKIIv92Rfs75HuT8cAQPZVwEDyXvM+kY9xhANHA8BA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nfqMcYooHjIWAg+ypgKIuAoQi+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0cDwEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrG/UYwzRwPEQMJB9FTCURcBQBF/uyb5mfY9yfzgCBrKvAgaS95j1jXqMIRo4HgIGsq8ChrIIGIrgyz3Z16zvUe4PR8BA9lXAQPIes75RjzFEA8dDwED2VcBQFgFDEXy5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPpGPcYQDRwPAQPZVwFDWQQMRfDlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaT8L3L8ZJ+aHFT8QL2A8BA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nfqMcYovEEfHQxmylhhA3YEQED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxii8QR8bDGbKaGAYWcEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiMaT4BZJ7xEBA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nfqMcYooHjIWAg+ypgKIuAoQi+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0cDwEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrG/UYwzRwPEQMJB9FTCURcBQBF/uyb5mfY9yfzgCBrKvAgaS95j1jXqMIRo4HgIGsq8ChrIIGIrgyz3Z16zvUe4PR8BA9lXAQPIes75RjzFEA8dDwED2VcBQFgFDEXy5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPpGPcYQDRwPAQPZVwFDWQQMRfDlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUARf7sm+Zn2Pcn84AgayrwIGkveY9Y16jCEaOB4CBrKvAoayCBiK4Ms92des71HuD0fAQPZVwEDyHrO+UY8xRAPHQ8BA9lXAUBYBQxF8uSf7mvU9yv3hCBjIvgoYSN5j1jfqMYZo4HgIGMi+ChjKImAogi/3ZF+zvke5PxwBA9lXAQPJe8z6Rj3GEA0cDwED2VcBQ1kEDEXw5Z7sa9b3KPeHI2Ag+ypgIHmPWd+oxxiigeMhYCD7KmAoi4ChCL7ck33N+h7l/nAEDGRfBQwk7zHrG/UYQzRwPAQMZF8FDGURMBTBl3uyr1nfo9wfjoCB7KuAgeQ9Zn2jHmOIBo6HgIHsq4ChLAKGIvhyT/Y163uU+8MRMJB9FTCQvMesb9RjDNHA8RAwkH0VMJRFwFAEX+7JvmZ9j3J/OAIGsq8CBpL3mPWNeowhGjgeAgayrwKGsggYiuDLPdnXrO9R7g9HwED2VcBA8h6zvlGPMUQDx0PAQPZVwFAWAUMRfLkn+5r1Pcr94QgYyL4KGEjeY9Y36jGGaOB4CBjIvgoYyiJgKIIv92Rfs75HuT8cAQPZVwEDyXvM+kY9xhANHA8BA9lXAUNZBAxF8OWe7GvW9yj3hyNgIPsqYCB5j1nfqMcYooHjIWAg+ypgKIuAoQi+3JN9zfoe5f5wBAxkXwUMJO8x6xv1GEM0cDwEDGRfBQxlETAUwZd7sq9Z36PcH46AgeyrgIHkPWZ9ox5jiAaOh4CB7KuAoSwChiL4ck/2Net7lPvDETCQfRUwkLzHrG/UYwzRwPEQMJB9FTCURcBQBF/uyb5mfY9yfzgCBrKvAgaS95j1jXqMIRo4HgIGsq8ChrIIGIrgyz3Z16zvUe4PR8BA9lXAQPIes75RjzFEA8dDwED2VcBQFgFDEXy5J/ua9T3K/eEIGMi+ChhI3mPWN+oxhmjgeAgYyL4KGMoiYCiCL/dkX7O+R7k/HAED2VcBA8l7zPpGPcYQDRwPAQPZVwFDWQQMRfDlnuxr1vco94cjYCD7KmAgeY9Z36jHGKKB4yFgIPsqYCiLgKEIvtyTfc36HuX+cAQMZF8FDCTvMesb9RhDNHA8BAxkXwUMZREwFMGXe7KvWd+j3B+OgIHsq4CB5D1mfaMeY4gGjoeAgeyrgKEsAoYi+HJP9jXre5T7wxEwkH0VMJC8x6xv1GMM0cDxEDCQfRUwlEXAUIRsIJDs6yj3h5Pr32pOkqzt3uNHIGAg+ypgqMsYooHjIWAg+ypgKIuAoQjZQCDZ11HuDyfXv9WcJFnbvcePIBuQJPuKeowhGjgeAgayrwKGsggYiqBBSPZ17wZhrp9kX0e570I2ILd+/UyytgKGuowhGjgeAgayrwKGsggYipANhK3mJMna7t0gzPWT7Oso910QMJB9FTDUZQzRwPEQMJB9FTCURcBQhGwgbDUnSdZ27wah5iDZ16zvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4jGk/D9i3FSfmjxE/EC9kPAQPZVwFAWAUMRBAxkXwUMJG8163uU+y4YQ8i+Zn2jHmOIxhPw0cVspoQRNmBHBAxkXwUMZREwFEHAQPZVwEDyVrO+R7nvgjGE7GvWN+oxhmg8AR9bzGZKKGDYGQED2VcBQ1kEDEUQMJB9FTCQvNWs71Huu2AMIfua9Y16jCEaT4JbJL1HBAxkXwUMZREwFEHAQPZVwEDyVrO+R7nvgjGE7GvWN+oxhmjgeAgYyL4KGMoiYCiCgIHsq4CB5K1mfY9y3wVjCNnXrG/UYwzRwPEQMJB9FTCURcBQBAED2VcBA8lbzfoe5b4LxhCyr1nfqMcYooHjIWAg+ypgKIuAoQgCBrKvAgaSt5r1Pcp9F4whZF+zvlGPMUQDx0PAQPZVwFAWAUMRBAxkXwUMJG8163uU+y4YQ8i+Zn2jHmOIBo6HgIHsq4ChLAKGIggYyL4KGEjeatb3KPddMIaQfc36Rj3GEA0cDwED2VcBQ1kEDEUQMJB9FTCQvNWs71Huu2AMIfua9Y16jCEaOB4CBrKvAoayCBiKIGAg+ypgIHmrWd+j3HfBGEL2Nesb9RhDNHA8BAxkXwUMZREwFEHAQPZVwEDyVrO+R7nvgjGE7GvWN+oxhmjgeAgYyL4KGMoiYCiCgIHsq4CB5K1mfY9y3wVjCNnXrG/UYwzRwPEQMJB9FTCURcBQBAED2VcBA8lbzfoe5b4LxhCyr1nfqMcYooHjIWAg+ypgKIuAoQgCBrKvAgaSt5r1Pcp9F4whZF+zvlGPMUQDx0PAQPZVwFAWAUMRBAxkXwUMJG8163uU+y4YQ8i+Zn2jHmOIBo6HgIHsq4ChLAKGIggYyL4KGEjeatb3KPddMIaQfc36Rj3GEA0cDwED2VcBQ1kEDEUQMJB9FTCQvNWs71Huu2AMIfua9Y16jCEaOB4CBrKvAoayCBiKIGAg+ypgIHmrWd+j3HfBGEL2Nesb9RhDNHA8BAxkXwUMZREwFEHAQPZVwEDyVrO+R7nvgjGE7GvWN+oxhmjgeAgYyL4KGMoiYCiCgIHsq4CB5K1mfY9y3wVjCNnXrG/UYwzRwPEQMJB9FTCURcBQBAED2VcBA8lbzfoe5b4LxhCyr1nfqMcYooHjIWAg+ypgKIuAoQgCBrKvAgaSt5r1Pcp9F4whZF+zvlGPMUQDx0PAQPZVwFAWAUMRBAxkXwUMJG8163uU+y4YQ8i+Zn2jHmOIBo6HgIHsq4ChLAKGIggYyL4KGEjeatb3KPddMIaQfc36TkbZA8DzImAg+ypgKIuAoQgCBrKvAgaSt5r1Pcp9F4whZF+zvpNR9gDwvAgYyL4KGMoiYCiCgIHsq4CB5K1mfY9y3wVjCNnXrO9klD0APC8CBrKvAoayCBiKIGAg+ypgIHmrWd+j3HfBGEL2Nes7GWUPAM+LgIHsq4ChLAKGIggYyL4KGEjeatb3KPddMIaQfc36TkbZA8DzImAg+ypgKIuAoQgCBrKvAgaSt5r1Pcp9F4whZF+zvpNR9gDwvAgYyL4KGMoiYCiCgIHsq4CB5K1mfY9y3wVjCNnXrO9klD0APC8CBrKvAoayCBiKIGAg+ypgIHmrWd+j3HfBGEL2Nes7GWUPAM+LgIHsq4ChLAKGIggYyL4KGEjeatb3KPddMIaQfc36TkbZA8DzImAg+ypgKIuAoQgCBrKvAgaSt5r1Pcp9F4whZF+zvpNR9gDwvAgYyL4KGMoiYCiCgIHsq4CB5K1mfY9y3wVjCNnXrO9klD0APC8CBrKvAoayCBiKIGAg+ypgIHmrWd+j3HfBGEL2Nes7GWUPAM+LgIHsq4ChLAKGIggYyL4KGEjeatb3KPddMIbs5xd+7W+a/w3+Kn/0j/v0zXXu5Zd+03e/23bs99Y8rGfWd7Kc3258dDHe1w+9/O31xHKxfKwHwDMgYCD7KmAoi4ChCAKGGv6S3/jzX+rox3/6j/3h3/aHfv3mPPf6PrbB96uAgeStZn2Pct8FY8h+/vSf96tf/p9+q1vr3Mt5X3/SR754cx7WM+s7Wc5vNz62mJ/dW8hlYz0AngEBA9lXAUNZBAxFEDCcNhvu4S/8lV+1Oc8pf+7Xf9m7ZR/RrI/t5/r+8d/9yz8wLdYfoUBM+6Kv+IIPTHuN57bBmgoYeukXyXyfZn0v53A3jCH7+fFv+b6Xxv2Wy6F/8dN+4mdvTn/fNTuPFx/7hu/YnIf1zPpOlvO7N3lFwSUfdcXAWwUMMf93Lf7gYq4jroaI11wNAdyDgIHsq4ChLAKGIggYThuN++UQvfiZn/2hzXlOmQ3/1y53ynPN//h7Tgvnaa9RwNBPAUMvo/G3HPab3VrnXs776hfJNc36Xs7hbhhD3sbl0KtN7m7Wd/JS9PsyN/zPWTlg+PbFXO6Un1gEcAsCBrKvAoayCBiKIGA474c/8tnvPsO/9fd98+Y8a3/d7/gl75Z57ZUPp7z2CoYv/wVf9IFpr3HvgCH2LdYd+7o1nY9XwNBLv0jm+zTrezmHu2EMeRuXQ/+igIF7mvWdvBT9vswN/9c07W/lLQKG71/83sUIET4cLyx8aPEbF3N9tz4TAoCAgeyrgKEsAoYiCBjOO98mKW57tDXP2mykh496lsH7uLpg72189Is+7936t6bz8QoYjuNyOl7UMOSjzPp+KfadMIa8jcuhN15wd7O+k5ei35cjBAznmK9ueB/vH+iHgIHsq4ChLAKGIggYznvLbZLyaoK4+mFr+i0KGHiLAobjuJyOFzUM+Sizvl+KfSeMIW/jcuivGi/i2S0x35f/2t/z8ve4Gipfi6ul8vUwrpKK9X3ocz/ybv1pvH7uSqa4Oivnjauf1tPz+TO5v1/znX/8ZV9iH3K52K6rpZ7LrO9kOU97c/SA4X2/f6AfAgayrwKGsggYiiBguOzcGL90m6T59khx9cPWPLcoYOAtChiO43I6XhQw8FFmfb8U+04YQ97G5dBfNV7kfBEk/LQv+cS7v6fR5I/5Yvp62pantjcvn+ucjddy+Zg3Q44tYz/Xy/NtzPpOlvOzN/c02GP+9YOT48/x2ql1zds7RdzK6AcWc774cz4jIV977b6eItb76HUCx0LAQPZVwFAWAUMRBAyX/RX/5Ne9+xxfuk3Sqdsjfc+/8m0vy0aTPa6EyHnCuOLhi77iC8429S81//OqiXP7F/sQ65mfKxF/jvcX0y9tIwKT2P95+TRez/XMzleAnHMruIn9zW3O857aFj9VAcNxXE7Hi+cahvOvhOPv8SvguWkYvwCO13La53/8G17WN/9KOIzm3uf8zK/8wK+X1176RXIsH9OymfhzfvMfedmXuXF4aRvc16zv5VzshjHkbVwO/YvXBgzzVQlRp1G38xUDUe85LsS0qNs0rj6Yx5Ctqwxivpx+LmCYx4fY9xhbYtmYPk+L19br4Ps36ztZzs3ezA3/1zTYf83iu8/PCeMZB2suBQwRTszrmJ2nPSoMyCAjghEAtyBgIPsqYCiLgKEIAobLRqN7OVQvXrpNUjb6owk+vz43788Z883LbS2/1fzPaevtptHAz33bMpaLcCL/vt5G/H2e/5Tr7cd2t+Zbu95eLLcOYtZG0BHnZl6OH1TAcByX0/HiuYbh3MSLAGHrVibZmMuG3iW3moHhpYZh7GdOy9ufnNJtT97GrO/lHOyGMeRtXA79i9cGDGE08CNI2JrvkhEgntvmpfFiPR5tPbQ+xomcHkHHejrfv1nfyXJu9uaWgOHji7lMPDz5o4tJTJuvPoi/z5wLGObQIsKEeBBzEMvEg5pzWnhvwBD7leuMBzzP7wHAaxAwkH0VMJRFwFAEAcN1xhUGy+F68dRtkubbI61/YR+/xI+GeVzhEH+Ohnoajf25+b/1YOh7AoZows/N+pgn9jXWE/uyFTyst5EBRRyH2Jfc91zHvP71e8/55isf5uXDef7Y39yn+G+sL4OEODbzsYh1zsvygwoYjuNyOl68NmDIcCGahnGlQhi/Mo5GYMybvzqOXyvHn2PZNOadfy2cy5za1rmAYV5PbCu3MV9ZEfPklRV8f2Z9L+dgN4whb+Ny6F98TcBwa7iQZs3H2LOedmm8mAOGrXAhzfEkxq6t6Xy/Zn0ny7nZm7nhf8powM/kLZGiQb9FBAM5T4QNM+cChkvrvecKhvlWSLOxrQ8vArgVAQPZVwFDWQQMRRAwXGc0uZfD9eKp2xDNt0d67S/r5/VvXcVwT8AQAUBOj4BgPT32NZbLecKtbZxzvhXS1j6E8za2pqfzcTwV5szvKcKSrXkoYDiSy+l48dqAIYxG362N+/nXwlsNwWsDhjCagjH/ep45ZDjXVOQ+Zn0vx383jCFv43LoX7w2YLg03zXONb+e9pqAYWusSM9tg+/frO9kOS9789qAIX7pn6+vr06Yidsj5Xx5JUJwKmC4dr05z6MChjCCjdeuD0AiYCD7KmAoi4ChCAKG681f1Z+6TVJO32riX2MsG24FGPcEDPOVA1tXR6RzAPDagCHM5U9dVXBNwBBhR84TQcPWPOm18x1ZAcNxXE7Hi+cagXMTL37he+9VAbmuuKJhPe01AcOpX0bPz3GIsGFrHu5n1vdy/HfDGPI2Lof+xUvBwbXzpXE1Uz67Zb46ae16OQFDT7O+k+W87M3c8L+myT4HB+c4td759Zn59TmQWJPzXLOv54irFuK9RHiS68wHSQN4DQIGsq8ChrIIGIogYLje+Zf16wb8udsjXWsuvxUS3BMwnJs2e2kbl7wUIFwTMMzH8dKVCbm+S+/ryAoYjuNyOl481wi81MR7ree2+ZqAYT1tNuc59764j1nfy/HfDWPI27gc+qvq6tr5wggWcv5LrpcVMPQ06ztZzsvenAoCTjE/J+Ecp9Y7vz5z7Xpznmv29RoiaMiQYX0rKADXIGAg+ypgKIuAoQgChuudm9/rX81n+BBXMcyvr43Gfcw7N9vXbjXMbw0Y5ocsn7q1U3pNwBBXQMR6Yjt5xcaWW8vO73lrejjvw7VuHS9+UgHDcVxOx4vnGoG3BAyxTFw9MDfu1m5tU8BQ36zv5fjvhjHkbVwO/VV1de18cQuznDduvRa3UFtfIXWu5gUMPc36TpbzsjengoBT7BUwxG2R8vX3cQXDzLcv5no97Bl4LQIGsq8ChrIIGIogYHid2VRfBwn5+qnb9Ww95+CUWw3zWwOGmDenxTrmaWsvbSOChZx+yfWy4V4Bw6ljTgHDkVxOx4vnGoGvCRiiOTg36865tU0BQ32zvpfjvxvGkLdxOfRX1dW188Ut12K+cw9kP1fzAoaeZn0ny3nZm1NBwCmunf/UrZTm5Wfm1/d4BsM5BAzAPQgYyL4KGMoiYCiCgOF1zk32vIXPNbf1iecy5Dzx5635cvpeAcM9VzDMD1WOZyzEbaDWD7K+FCC8NmDYep98nQKG47icjhfPNQJfEzB8zs/8ynfzxp+/9Ju++1Pmyelb2xQw1Dfrezn+u2EMeRuXQ39VXT1yvnM1L2DoadZ3spyXvbk2MEji6oK8pdD3xgsbxDzx4OSteU4FDMGl9X7XYi77qIBh3le3SAJuQcBA9lXAUBYBQxEEDK9zvuVQ/nL+0u2R4rZCucxWeJCem+fWgOHStNlz24gHW8fr8R7XwUJ6KUC4ND2cw5oINbbm4fUKGI7jcjpePNfguzZgiAe1XrO+c/MIGOqb9b0c/90whryNy6G/qq5eO19cybA1PW6ZlPOE6+kChp5mfSfLedmb1wYMwXybpAgD5l/9x9UHP7AY06JhH884mDkXMMwBQmwjb5UUy+Q602v3NeaLACGuUlgvM+9rbhPAaxEwkH0VMJRFwFAEAcPrnZvt8fdLt0earyKIJv7WPGHO8+iAIfc3PBUOxOtxZULOt95Gvn4upHhEwHBtGMPrFDAcx+V0vHiuEXhtwHDtfDnP1jYFDPXN+l6O/24YQ97G5dBfVVfXzhfPacl5488RUsbr8d95WrpeXsDQ06zvZDkvezM3/K9t2gdzGLBlhAtbtzqat7cmAoV1kDAbIcH3jz9fu68RfqzXs2WsG8AtCBjIvgoYyiJgKIKA4fXOt0nKqxfCuLpha/45YDgVQszr2Wqs3xMwrPd3PX0dLoSnAoYIK+bX07hl0rz81jzzfpw6VuEcRMR6t+ZJz62HAoYjuZyOF881Am8JGKI5uDXP3DTc2qaAob5Z38vx3w1jyNu4HPqr6ura+SJIiOcv5PxrY9p827X18gKGnmZ9J8t52Zu5Ab++2uASn1iMKxjy1kZhBATRrD+1rtzeqdsRRcgQVxLkbYvC2EYGCvHneO01YUjsyxxOpLGNCEpesy4AawQMZF8FDGURMBRBwPB651/Zp6ca7+l8FUHc+ievJIhG/txQDx8dMMT+5lUWYT4DItYT681p8zzrbcwBSPw51pnrnqel87LpHEJEoJH7MK8vnG9DFcZ7imVj3jCWi7AiQ5H1vvJHFDAcx+V0vHiuEXhtwBDmQ1vDL/za3/Tuwa2xjrmJF25tU8BQ36zv5fjvhjHkbcww4FSAmOZ8n//xb9icPhtjRMw3jx0f+tyPvNR/TItxJF6Lda6X/fi3fN+7Zbae9zIvm1dHbJkhxtY2+P7N+k6WcwMAz42AgeyrgKEsAoYiCBhuc/2L/0sPUJ6fLbBlrC/X+eiAIYwG/RwgrI0m/7yP622sQ4q1MW1+kPW8bBqhyhy0zK63F/tybnuzrmI4rYDhOC6n48VHBQzR5Mt5t4zGYRh/FjD0NOt7Of67YQwh+5r1nYyyB4DnRcBA9lXAUBYBQxEEDLc5/xo/GuHzL/BPGY3waOTPjfP8dX5MzwZ9/He9bFz1cG5buc5zQUdebTBvP7aVzf35yoGtpn0EBLH+OSSIUCTCj5g27+N62XS9D/Hf2Ie8omM21xnHKLcXxvbjtdjuNcf9yAoYjuM1v0i+9CvhtTF/rC/XHUajPx7YGtPz18Lx361lz20rl42QYj1tNre9tQ3ua9b3cvx3wxhC9jXrOxllDwDPi4CB7KuAoSwChiIIGMi+ChhI3mrW9yj3XTCGkH3N+k5G2QPA8yJgIPsqYCiLgKEIAgayrwIGkrea9T3KfReMIWRfs76TUfYA8LwIGMi+ChjKImAogoCB7KuAgeStZn2Pct8FYwjZ16zvZJQ9ADwvAgayrwKGsggYiiBgIPsqYCB5q1nfo9x3wRhC9jXrOxllDwDPi4CB7KuAoSwChiIIGMi+ChhI3mrW9yj3XTCGkH3N+kY9xhANHA8BA9lXAUNZBAxFEDCQfRUwkLzVrO9R7rtgDCH7mvWNeowhGjgeAgayrwKGsggYiiBgIPsqYCB5q1nfo9x3wRhC9jXrG/UYQzRwPAQMZF8FDGURMBRBwED2VcBA8lazvke574IxhOxr1jfqMYZo4HgIGMi+ChjKImAogoCB7KuAgeStZn2Pct8FYwjZ16xv1GMM0cDxEDCQfRUwlEXAUAQBA9lXAQPJW836HuW+C8YQsq9Z36jHGKKB4yFgIPsqYCiLgKEIAgayrwIGkrea9T3KfReMIWRfs75RjzFEA8dDwED2VcBQFgFDEQQMZF8FDCRvNet7lPsuGEPIvmZ9ox5jiAaOh4CB7KuAoSwChiIIGMi+ChhI3mrW9yj3XTCGkH3N+kY9xhANHA8BA9lXAUNZBAxFEDCQfRUwkLzVrO9R7rtgDCH7mvWNeowhGjgeAgayrwKGsggYiiBgIPsqYCB5q1nfo9x3wRhC9jXrG/UYQzRwPAQMZF8FDGURMBRBwED2VcBA8lazvke574IxhOxr1jfqMYZo4HgIGMi+ChjKImAogoCB7KuAgeStZn2Pct8FYwjZ16xv1GMM0cDxEDCQfRUwlEXAUAQBA9lXAQPJW836HuW+C8YQsq9Z36jHGKKB4yFgIPsqYCiLgKEIAgayrwIGkrea9T3KfReMIWRfs75RjzFEA8dDwED2VcBQFgFDEQQMZF8FDCRvNet7lPsuGEPIvmZ9ox5jiAaOh4CB7KuAoSwChiIIGMi+ChhI3mrW9yj3XTCGkH3N+kY9xhANHA8BA9lXAUNZBAxFEDCQfRUwkLzVrO9R7rtgDCH7mvWNeowhGjgeAgayrwKGsggYiiBgIPsqYCB5q1nfo9x3wRhC9jXrG/UYQzRwPAQMZF8FDGURMBRBwED2VcBA8lazvke574IxhOxr1jfqMYZo4HgIGMi+ChjKImAogoCB7KuAgeStZn2Pct8FYwjZ16xv1GMM0cDxEDCQfRUwlEXAUAQBA9lXAQPJW836HuW+C8YQsq9Z36jHGKKB4yFgIPsqYCiLgKEIAgayrwIGkrea9T3KfReMIWRfs75RjzFEA8dDwED2VcBQFgFDEQQMZF8FDCRvNet7lPsuGEPIvmZ9ox5jiAaOh4CB7KuAoSwChiIIGMi+ChhI3mrW9yj3XTCGkH3N+kY9xhANHA8BA9lXAUNZBAxFEDCQfRUwkLzVrO9R7rtgDCH7mvWNeowhGjgeAgayrwKGsggYiiBgIPsqYCB5q1nfo9x3wRhC9jXrG/UYQzSehO9fjJPyQ4ufiBewHwIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0noCPLmYzJYywATsiYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xROMJ+NhiNlNCAcPOCBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNF4Etwi6T0iYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsAoYiCBjIvgoYSN5q1vco910whpB9zfpGPcYQDRwPAQPZVwFDWQQMRRAwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChrIIGIogYCD7KmAgeatZ36Pcd8EYQvY16xv1GEM0cDwEDGRfBQxlETAUQcBA9lXAQPJWs75Hue+CMYTsa9Y36jGGaOB4CBjIvgoYyiJgKIKAgeyrgIHkrWZ9j3LfBWMI2desb9RjDNHA8RAwkH0VMJRFwFAEAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmAoi4ChCAIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUBYBQxEEDGRfBQwkbzXre5T7LhhDyL5mfaMeY4gGjoeAgeyrgKEsX78Y5232Tyz+hEU8EQIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXAUJbPWozztvbPLX5iEU+CgIHsq4CB5K1mfY9y3wVjCNnXrG/UYwzRwPEQMJB9FTCUJq5YiHO35Z9cjKDhRy3iDREwkH0VMJC81azvUe67YAwh+5r1jXqMIRo4HgIGsq8ChtJ83uKfX4zzd8q4ouE7Fz+2iDdAwED2VcBA8lazvke574IxhOxr1jfqMYZo4HgIGMi+ChjK87MW/8JinMNLRtjw3Ytfs+hZDe8JAQPZVwEDyVvN+h7lvgvGELKvWd+oxxiigeMhYCD7KmBoQTyP4fcvxnm81r+8+AOLcXVD3ErpcxexAwIGsq8CBpK3mvU9yn0XjCFkX7O+UY8xRAPHQ8BA9lXA0IqvWjz3XIZLxu2W/uDib1mMqxziFky4EwED2VcBA8lbzfoe5b4LxhCyr1nfqMcYooHjIWAg+ypgaMmXLP7hxbhKIc7tPf7FxbjS4fcu/obFCB6+YPHHLOIKBAxkXwUMJG8163uU+y4YQ8i+Zn2jHmOIBo6HgIHsq4ChNZ+x+PWLf3TxLy3GeX6kccVDXDER4UNc9fBLF3/24ucvxraxIGAg+ypgIHmrWd+j3HfBGEL2Nesb9RhDNHA88ssTyb6OckdfPm0xbqEUD3n+s4txzvc2Qo3Y1h9b/D2L37EYV0H8osWvXowrLeLZD63DiK16I9nLUe4PJ5sHJPs6yn0XtrZHspeoxxiigeOx9UWKZC9HueM4/NTFr1387Yt/cvERt1O617gdUwQS4b+2GMFEPBPidw2/bfFbF795MUKK8MsWI6iYjSsoIrg45U9YfG9s1RvJXo5yfzhbjQSSvRzlvgtb2yPZS9RjDNHA8cgvT7/yX/oKks3cuzmCMsQVDj9r8ZctRugQDf6/sBifje5GaAEAAAAApRg9axRinDrgeAgYyL4KGHCBn7IYVwlE8PCdi3FFwZ9ZnBv01RUwAAAAACjH6FmjEOPUAcdDwED2VcCAO4jbLEX4EA+Sjoc8x8Oe48qHuMXRHg+V3ksBAwAAAIByjJ41CjFOHXA8BAxkXwUM2JG47VKEEPFchE8sxlUQ0cyP5yn8gcVnCSMEDAAAAADKMXrWKMQ4dcDxEDCQfRUw4In4rMV4CPPnLeYDm79mMR7mHFdJRBAQxjMi8sHP8SDotX96MR8WvWU8TFrAAAAAAKA0o2eNQoxTBxwPAQPZVwEDAAAAAAD1GD1rFGKcOuB4CBjIvgoYAAAAAACox+hZoxDj1AHHQ8BA9lXAAAAAAABAPUbPGoUYpw44HgIGsq8CBgAAAAAA6jF61ijEOHXA8RAwkH0VMAAAAAAAUI/Rs0YhxqkDjoeAgeyrgAEAAAAAgHqMnjUKMU4dcDwEDGRfBQwAAAAAANRj9KxRiHHqgOMhYCD7KmAAAAAAAKAeo2eNQoxTBxwPAQPZVwEDAAAAAAD1GD1rFGKcOuB4CBjIvgoYAAAAAACox+hZoxDj1AHHQ8BA9lXAAAAAAABAPUbPGoUYpw44HgIGsq8CBgAAAAAA6jF61ijEOHXA8RAwkH0VMAAAAAAAUI/Rs0YhxqkDjoeAgeyrgAEAAAAAgHqMnjUKMU4dcDwEDGRfBQwAAAAAANRj9KxRiHHqgOMhYCD7KmAAAAAAAKAeo2eNQoxTBxwPAQPZVwEDAAAAAAD1GD1rFGKcOuB4CBjIvgoYAAAAAACox+hZoxDj1AHHQ8BA9lXAAAAAAABAPUbPGoUYpw44HgIGsq8CBgAAAAAA6jF61ijEOHXA8RAwkH0VMAAAAAAAUI/Rs0YhxqkDjoeAgeyrgAEAAAAAgHqMnjUKMU4dcDwEDGRfBQwAAAAAANRj9KxRiHHqgOMhYCD7KmAAAAAAAKAeo2eNQoxTBxwPAQPZVwEDAAAAAAD1GD1rFGKcOuB4CBjIvgoYAAAAAACox+hZoxDj1AHHQ8BA9lXAAAAAAABAPUbPGoUYpw44HgIGsq8CBgAAAAAA6jF61ijEOHXA8RAwkH0VMAAAAAAAUI/Rs0YhxqnDk/D9i3FSfmjxE/EC9kPAQPZVwAAAAAAAQD1GzxqFGKcOT8BHF+OEpBE2YEcEDGRfBQwAAAAAANRj9KxRiHHq8AR8bFHA8B4RMJB9FTAAAAAAAFCP0bNGIcapw5PgFknvEQHDeX/hP/VFGXbdZCy/td5qfuWv+pt/+K//4s/64f/yp/2V797bZ/51n/7DX/jzPvzDv/i7/puby/DtFTAAAAAAAFCP0bNGIcapA46HgOG8v+C/+4XvGuq3GMtvrbeKv/R3/20vQcLWe5v9Ob/eZ+gZFTAAAAAAAFCP0bNGIcapA46HgOG80WD/2Nf99Zv+1Z/149412Lemh7H81nr3NvctrjrYmn6tc7jwOV/wGS9XMkRoEoHC3/wVn/3uioZ4r1vL32tsL7ff5WqQ96mAAQAAAACAeoyeNQoxTh1wPAQMtxsN9+UQvrg1/S3N/Yp93Jp+jREiXFpP3B4pQoy//Rv/xs3p9xrBRe5D9atB3kIBAwAAAAAA9Rg9axRinDrgeAgYbrd7wDA399/qFkgChvsUMAAAAAAAUI/Rs0YhxqkDjoeA4XaPFDC8VXNfwHCfAgYAAAAAAOoxetYoxDh1wPEQMNzuawKGaI7HMwvm5zbEn+O1uM3Q1jJpPgdifh5CLjs33efnFZwynpkwr/ucc3M/trU1zzXG/sctlObjFcbfY5+3lpmP0yn3ui1TJwUMAAAAAADUY/SsUYhx6oDjIWC43WsDhi/8eR9+N98pTzXa48HG+SDlU+a80XDfmr52Xv85I/iYl4vAYWu+c8b+XwoLIjhZPwz70nsOb9mfoylgAAAAAACgHqNnjUKMUwccDwHD7V4TMMxXAcTDkKPhntPiuQbzVQlbzznIbUTDfZ4ezf9Y93xFQjTp44qGMNcZ68/Xwnn717gORyIsOBWGrI39yaAg/hvLZZCQ+5/rjf2cl439jP2NKydynghQ5veyDiX4qQoYAAAAAACox+hZoxDj1AHHQ8Bwu5cChvkKgAgXtuaJJnn+wn/dZA9z+Wj0r6edM5eLfdya/hq3rsCIfb50i6I5HDgVbMxXXWwFLHMIEaHCejrPK2AAAAAAAKAeo2eNQoxTBxwPAcPtXgoY5sb8uecszE309Xz5+lsGDGE09+f3m0bQsBUMRHCS81x6fsO5+QQM9ylgAAAAAACgHqNnjUKMUwccDwHD7V4KGPL2R1tXJsxG4zzXs26iz88viGb7tbcFymUeFTCkp4KG9W2TInTIaVsBxGyub2tfBQz3KWAAAAAAAKAeo2eN/397d2ztSlalAdgfB2Mc7DZw28OBCHDHmQzIYDIgBWIgBHIgBWLoFEB/r316NocqSfdePfS29H1r/Wv1lapUJZ3Xzv5VqkFq6eD9KBg+n1sFw3ruI9mH6PlpoX7D4/x3vul/614Ka/tHFwwrOc9VoKz0qy96MXBvFAyPj4IBAAAA5qmZNYPU0sH7UTB8Po8uGFIeHBUHGdz3+xms5PhnRUPf5uj5RyRXU/SSIWXAeu4zBYOfSHp8FAwAAAAwT82sGaSWDt6PguHzubdgeNSQP0VDborcfzbprJR49LHP0n8KqR/rUcWAguFrUTAAAADAPDWzZpBaOng/CobP51bBsL7dnxLg6PmvJEXDOvbRN//Xc9+6YMjQ/+hYvXjIufZ9PhIFw9eiYAAAAIB5ambNILV08H4UDJ/PrYKh/6zRtxiOrwLjqERYx/3WBUNu7ryOlTJgPZ6rLR5xDgqGr0XBAAAAAPPUzJpBaul4cz9c8tMl+QfxtzzwDhQMn8+tgqF/uz9lQO5ZcLRdkuf6TZKTDO9zjP3xJNuvmz8fDfAfcfVE9s2A/+j4Sd7fOodk/6mm/vnkvfTn9pzdS+JRV0K8axQMAAAAME/NrBmklo439/tLfhlm5oF3oGD4fG4VDMlvfvfrX7bJMP63//PDz4P5lQzNs02e24uC9e39fb8M6/vNlY+G9/3qibx+9suw/ujnlM6y9k9yvJxDzinpx0/y3L5/SoO+Td5fznW9j5xP9luvlcf21+hXQuRzWPvnHLL/vr38axQMAAAAME/NrBmklm6sHy/50yX51v0vw7j6+8+X/O8l3PY9Fgz/d0k/p6zzvbL2a7/8GzmkYPh87ikYcqVBLxmuJdv1fTNc71cIHOWsMNiH+z1H2x9lLxHOclQurKQEuPUeVs6uYjj7/FIyHG0v/x8FAwAAAMxTM2sGqaUbKQPnfxm6neRtfvLnC77HguGvl/RzSlI63KPvm/d2SMHw+azB969+/V+Hz/fkW/cpA7Jt9llJSZHHz35CKAXFfsVABvbrqoSjfVbyfB/O59jXyoCjrKse9rIhf+e1zkqBnryHXKnRC5kk55PH8v7OfoYpWZ9B/+yy3z3HfvcoGAAAAGCemlkzSC3dOP0b6klKhAyfk6MrGrhuSsGQ+0T89yW3KBhE3jwKBgAAAJinZtYMUks3yj4MP/sZpNy4OEXE33/+i2umFAzJPT+VpGAQefMoGAAAAGCemlkzSC3dKP3qhfw3X/e9Fwz7FSspj65RMIi8eRQMAAAAME/NrBmklm6UPjy+9zf5ue57LxhyfrkSZf39l0uuUTCIvHkUDAAAADBPzawZpJZulEcWDD9ektfIa+b3/dfrJnnsj5fc+s3/db+HPvTOUDt/r9fKa+db+EfD7vzEU982g/Rse+u463z7cfvPQq3Xy/nlfVzz0YIh57Y+t75f/s7j99wn4Zb+2jm/P7S/12Nn9n0PKRhEXjcKBgAAAJinZtYMUks3yj6M/+wwex+OnyWD/BQRZ9Z2eb2cy/5zPnvWPSOy7bVz+MhxI4P9vv+eFA1nn9VHCoac/17G7MnzKQS+4qgk6I+t933kaN9/o2AQed0oGAAAAGCemlkzSC3dKBlwr+Fxcm1wfmYfqOc1cvPgDOmTXmIk124U3V8jWX9nyJ3XSuGwD+QzfO/b5nhr277dPcfNcfp+2We9lz5oT3LMI/cWDPtn34+1XzmRXCtIbjkqCfbzXGXNTsEg8uZRMAAAAMA8NbNmkFq6cfpwPskAP0Pue4uG/JRQ9slw/OyGwRmO92OcfSO/b5McfXs/r7WXDGvbfQi/H/dsiN63WTn6KaS9FDh6vXsKhnxOfZujY+1XcFwrSG45KwnueX0Fg8ibR8EAAAAA89TMmkFq6cbJIHsvGZJVNJyVBh/Vh9l53SP78c++tZ8y495t+3Hz30f6ayVnRUT01zu6iuGegqG/Rt7LNf1Khs/+VNJZSbAXHUfromAQefMoGAAAAGCemlkzSC3dSCkZ9qF9T5776E8n7TK8Xq/Xb6bc9WNeu5nyPsQ/KxeiX3WQYfmR/lq3buKcIX/ffi9gbhUM+Rz787cKnJzP2vZWGXHmWknQ1yVFzb7OCgaRN4+CAQAAAOapmTWD1NKNlmF3/3Z9z7WrBO7RB9n3DPpPh9kX+5D+mj7w/+pxl2vb9+Mlu/782X0cunvO/5ZrJUE+y6zten4vMRQMIm8eBQMAAADMUzNrBqmlewkZOqcQ6IPn5J6SIftmEJ39e/qg+hGD/r7tNfcM6Ptr3Tpu9J8t2q946MdLdvks1nN5nf1z2tMLn7Pzv+VWSdCvkkj6VRW39v2ZgkHkdaNgAAAAgHlqZs0gtXQvJWVBfs5oDZiTsyF3fjro6F4OR3nEoL9ve00f+D/iuNGH7ikBun68ZJft+/Mfydn539LP9+z99dKk/4TVPfsqGEReOAoGAAAAmKdm1gxSS/eS9pJhv2/A2c8qneURg/6+7TV94P+I40Yfuv8nC4ZvcQ+GZb+3xNpOwSDy5lEwAAAAwDw1s2aQWrqXlEJhDZn3QfP+8zopIzKsztUPXR+sTy8Y+rf9cxPprh8v2d3zOTzaXSXBRd9undtd+yoYRF43CgYAAACYp2bWDFJL97LWkHkfNPdh+/5t/u5VCoYUJ9e278dLdv1KgXtu8vwId5UEF/u5528Fg8ibR8EAAAAA89TMmkFq6V7StSsY+uP7Tyd1UwqG/abNu1yx0Lff9eMdPb9/lvuVHt/CvQVD7DeVVjCIvHkUDAAAADBPzawZpJZujAzKc/XBrYFz9KHzT3mgWY8nP+aBE/0+Dt9zwZD3d+199IF7vxny0o+XHOk3w7521cej3FUSlL0Ayedxc18Fg8jrRsEAAAAA89TMmkFq6cboVxRkUH40PM6web/B8z4Q7z+RlCJil9foA+7key4YkgzV93srxH4z66Nz7MdLjuw3VL5VMqTw+EoR8ZGCIfq/jZ7TfRUMIq8bBQMAAADMUzNrBqmlG2MfhCcZrGcYnfRv2a8cFQj7TwalcMiAOtnLiZXvuWDo39jPZ7DeSy9Skj9dcmT/XM/sZUWOm8fW8ZKcbz+fa1dWXJPXWa9x63ON/GxTP+7NfRUMIq8bBQMAAADMUzNrBqmlGyX3GzgaJB/lbKAe+7B8T47Rv7X/PRcMOc+zYmTlqGhZ7i0YIp9p3/Za8hl+1kcLhtiLo6v7KhhEXjcKBgAAAJinZtYMUks3Tr6tnmFyhur7t/QzmM4Q/NrNm5c1mO+FxboCYN3IeD13NqBfV01ku1vHXK+Vc74m3/pf2+b8juS5lTVEz2fSB/N5jex/a0Cf817Hy/u5JdvnM+7HWslj+axyLl+5GfQqTHJeH7kKohcg+ZxP10TBIPK6UTAAAADAPDWzZpBaOgZaQ/Tk3m/40ygYRF43CgYAAACYp2bWDFJLx0AKhi9SMIi8bhQMAAAAME/NrBmklo6BFAxfpGAQed0oGAAAAGCemlkzSC0dAykYvkjBIPK6UTAAAADAPDWzZpBaOgZSMHyRgkHkdaNgAAAAgHlqZs0gtXQM9LdLsoA/XfJDHuBjFAwirxsFAwAAAMxTM2sGqaWD96NgEHndKBgAAABgnppZM0gtHbwfBYPI60bBAAAAAPPUzJpBaung/SgYRF43CgYAAACYp2bWDFJLB+9HwSDyulEwAAAAwDw1s2aQWjp4PwoGkdeNggEAAADmqZk1g9TSwftRMIi8bhQMAAAAME/NrBmklg7ej4JB5HWjYAAAAIB5ambNILV08H7WAFJEXjf1vzsAAAAwQM2sGaSWDt7P0TBSRF4r9b87AAAAMEDNrBmklg4AAAAAAJ6nZtYMUksHAAAAAADPUzNrBqmlAwAAAACA56mZNYPU0gEAAAAAwPPUzJpBaukAAAAAAOB5ambNILV0AAAAAADwPDWzZpBaOgAAAAAAeJ6aWTNILR0AAAAAADxPzawZpJYOAAAAAACep2bWDFJLBwAAAAAAz1MzawappQMAAAAAgOepmTWD1NIBAAAAAMDz1MyaQWrpAAAAAADgeWpmzSC1dAAAAAAA8Dw1s2aQWjoAAAAAAHiemlkzSC0dAAAAAAA8T82sGaSWDgAAAAAAnqdm1gxSSwcAAAAAAM9TM2vG+Mc//gk/lZAmfewQSAAAAABJRU5ErkJggg==" } }, "cell_type": "markdown", "metadata": {}, "source": [ "## Cross Validation\n", "Oftentimes, we want to compare how our model performs on different types of features. A simple way to do this would be to compare the test set scores of the model trained on each type of features.\n", "\n", "However, once again this can lead to overfitting, as the best performing set of features may simply just be well fitted to that particular test set; for other samples that set of features may score comparatively worse than the others. In order to avoid this problem, we use **cross validation**.\n", "\n", "Here, we use a strategy called k-fold cross validation. Essentially, this means that the model is trained k times on k different \"folds\" of the training data, with each fold having a separate training and validation set of its own. This means that for each set of features, k different models are trained, and the average score across these k validation folds are given as the final \"cross validation\" score.\n", "\n", "![cross_val.png](attachment:cross_val.png)\n", "\n", "The best performing set of features are then used to re-train the model with the full set of data, and tested on the test set.\n", "\n", "In summary:\n", "\n", "* Take out the test set - this is what the final best performing model will be evaluated on to give it its overall test score\n", "* Take our first feature set/model to be evaluated\n", "* Split the training data into a number of different \"folds\"\n", "* Train the data a number of times, using a different fold as the \"validation\" set each time. These act as pseudo-test sets.\n", "* Get the average of the validation scores across each fold - this gives a good idea to how well the model would generalise to a broad spectrum of data that it hasn't seen before.\n", "* Repeat this process for each model or feature set you want to evaluate.\n", "* Best performing feature set is then trained on the full set of training data, and tested on the held-out test set. This completely avoids overfitting and gives you a good idea of how the model will perform on data it has never seen before.\n", "\n", "---\n", "Below, we perform 5-fold cross validation across polynomial features ranging from 1 (i.e. the base set) to 5.\n", "\n", "The `KFold().split()` function in `sklearn` allows us to automatically split the data across a number of folds.\n", "\n", "it is trained on the training portion of each fold, and scored on the validation portion.\n", "\n", "For each `poly_degree`, we store the average of the score across all folds. The best performing set of features are then scored on the test set to get the final score of our model." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:47:19.691010Z", "start_time": "2017-10-02T20:47:19.568685Z" }, "collapsed": true }, "outputs": [], "source": [ "poly_degrees = []\n", "cv_folds = KFold(n_splits=)\n", "\n", "# set up the scores dictionary\n", "scores = {}\n", "\n", "# loop through each set of features\n", "for poly_degree in poly_degrees:\n", " # set up the transformer\n", " poly_transformer = PolynomialFeatures(\n", " degree=poly_degree, include_bias=False)\n", "\n", " # generate the new set of features\n", " X_train_poly = poly_transformer.fit_transform()\n", " \n", " # empty scores list\n", " scores[poly_degree] = []\n", " \n", " print('\\nscores for poly_degree={}:'.format(poly_degree))\n", " \n", " for i, (train_indices, validation_indices) in enumerate(\n", " cv_folds.split(X_train, y_train)):\n", " \n", " # get the X and y train and validation set for this fold\n", " X_train_cv = X_train_poly[]\n", " y_train_cv = y_train.iloc[]\n", " \n", " X_valid_cv = X_train_poly[] \n", " y_valid_cv = y_train.iloc[]\n", " \n", " # build the model and make predictions\n", " oil_use_model = LinearRegression(fit_intercept=True)\n", " oil_use_model.fit(, )\n", " \n", " y_hat = oil_use_model.predict()\n", " \n", " # score it (actual vs predicted)\n", " fold_score = r2_score(, )\n", " \n", " # store the score\n", " scores[poly_degree].append(fold_score)\n", " \n", " print('fold {}: {:.2f}'.format(i, fold_score))\n", " \n", " average_score = np.mean(scores[poly_degree])\n", " print('average across folds: {:.2f}'.format(average_score))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training and Testing Model on Best Feature Set\n", "\n", "As can be seen, the best set of features was for `poly_degree`=3. In this case, we now train a model on the full set of training data (as opposed to three separate folds), and test on the held-out test set. This should give us a very confident estimate of how the model will perform on new, unseen data.\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:47:21.760731Z", "start_time": "2017-10-02T20:47:21.743688Z" }, "collapsed": true }, "outputs": [], "source": [ "poly_transformer = PolynomialFeatures(degree=3, include_bias=False)\n", "# generate the new set of features\n", "X_train_poly = poly_transformer.fit_transform()\n", "X_test_poly = poly_transformer.transform()\n", "\n", "final_oil_use_model = LinearRegression(fit_intercept=True)\n", "final_oil_use_model.fit(, )\n", "\n", "y_hat = final_oil_use_model.predict()\n", "\n", "score = r2_score(, )\n", "\n", "print('final score on test set:', score)\n", "\n", "print('intercept:\\n', final_oil_use_model.intercept_)\n", "print('weights:\\n', final_oil_use_model.coef_)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With an $r^2$ score of ~.94, the model performs very well and seems to be well suited to modelling on unseen data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Create vector of numbers from 1 to length of y_plot, to serve as the x-axis:\n", "nums = np.arange(len(y_hat))\n", "\n", "# sort the values in ascending order\n", "y_test_sorted = np.sort(y_test)\n", "y_hat_sorted = y_hat[np.argsort(y_test)]\n", "\n", "# difference between y_hat and y_test\n", "delta = abs(y_hat_sorted - y_test_sorted)\n", "\n", "# create the plot\n", "fig, ax = plt.subplots(figsize = (10,8))\n", "\n", "# Plot the delta as a line graph\n", "ax.plot(nums, delta, label = 'delta (|y_test - y_hat|)')\n", "\n", "# Plot y_hat\n", "ax.scatter(nums, y_hat, color='r', label='Predicted/Modelled Values')\n", "\n", "# plot y_test\n", "ax.scatter(nums, y_test, label='Actual Test Set Values')\n", "\n", "# Set up legend & title\n", "fig.suptitle('Plot of y_test vs. y_hat', fontsize='xx-large')\n", "ax.legend(fontsize='x-large', frameon=True, shadow=True)\n", "ax.set_xlabel('sample no.')\n", "ax.set_ylabel('predicted/actual heating oil usage (kWh)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Making future predictions\n", "If the $r^2$ score for the testing data is sufficiently good (~ > 0.75), we can assume the model generalises well to new data.\n", "\n", "\n", "With this in mind, we can feed the model new, unseen features which we do not have associated responses for. We can then use these features to make predictions using our model:\n", "
$\\hat{y}_{new} = w_0 + w_1x_{1 new} + w_2x_{2 new} + ... + w_mx_{m \\ new}$
\n", "\n", "At this point, since we have a good $r^2$ score on the test set, we can assume that our predictions on a new dataset are accurate. We can then use our predictions to draw conclusions from the data. For example, we can get an average of all $X$ and $y$ from our initial training/testing dataset, and compare to averages of $X_{new}$ and $\\hat{y}_{new}$.\n", "If the average of $\\hat{y}_{new}$ is higher than $y$, we can make a reasonable hypothesis of why this is the case by looking at the averages of $X$ and $X_{new}$, and comparing with the correlation matrix found earlier. For example, on the housing dataset, if the average age of houses on the new dataset is higher than on the one used for training and testing, we can infer that the heating oil usage should be higher as well.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "Here we'll import a second housing dataset, with houses that are not included in our previous training data.\n", "\n", "Note that there is no `oil_usage` column here, so we can make predictions for this based on our previous model!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:47:24.249467Z", "start_time": "2017-10-02T20:47:24.238463Z" }, "collapsed": true }, "outputs": [], "source": [ "house_data_2.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Input into existing model\n", "\n", "Since we now know that the model performs well on unseen data, we can say with some confidence that its predictions will be accurate.\n", "\n", "---\n", "Below, we make predictions from the new data.\n", "\n", "Note that we must first transform the features with a 3rd-degree polynomial, as before:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:47:26.499499Z", "start_time": "2017-10-02T20:47:26.493479Z" }, "collapsed": true }, "outputs": [], "source": [ "# Create a new matrix of features - insulation, age, temp, home_size\n", "X_new = house_data_2[]\n", "\n", "X_new_poly = poly_transformer.transform()\n", "\n", "# Create a vector of actual target values\n", "y_pred_new = final_oil_use_model.predict()\n", "\n", "print('predictions made')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Compare Predictions against Housing Dataset 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've now got predictions for all the new Data. We can save these predictions as a csv for pasting into excel, or manipulate them from within python.\n", "\n", "We can sum up and compare the total consumption from each dataset. Why is this higher or lower? In this case, it may be because of increased average temperature, more average occupants per house, or a better insulation rating. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2017-10-02T20:47:28.575409Z", "start_time": "2017-10-02T20:47:28.551340Z" }, "collapsed": true, "scrolled": false }, "outputs": [], "source": [ "# Can export y_pred_new for pasting into the existing Household Dataset1 csv,\n", "# for analysis in excel\n", "np.savetxt('house_dataset2_predictions.csv', y_pred_new)\n", "\n", "# Here, we'll perform analysis in python\n", "# First, we get the average of all features in dataset1 and _new\n", "# The (0) in .mean(0) means go along axis 0, i.e. the columns, instead of axis\n", "# 1, i.e. the rows\n", "print('means for Dataset1: \\n', house_data.mean(0), '\\n')\n", "print('means for Dataset2: \\n', house_data_2.mean(0), '\\n',\n", " 'estimated_oil_usage', np.mean(y_pred_new))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As can be seen, the estimated heating oil usage in dataset 2 is quite low.\n", "\n", "From the correlation matrix in Section 5 and partial regression plots in 6.3.1, we can see that insulation and average age are strongly positively correlated with Heating Oil usage. These are both lower on average for the houses in dataset2, so we see a lower heating oil usage. Temperature is negatively correlated, so it being higher on average in datset2 will also lower the heating oil usage." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" }, "nav_menu": {}, "notify_time": "10", "toc": { "colors": { "hover_highlight": "#DAA520", "navigate_num": "#000000", "navigate_text": "#333333", "running_highlight": "#FF0000", "selected_highlight": "#FFD700", "sidebar_border": "#EEEEEE", "wrapper_background": "#FFFFFF" }, "moveMenuLeft": true, "nav_menu": { "height": "386px", "width": "252px" }, "navigate_menu": true, "number_sections": true, "sideBar": false, "threshold": "3", "toc_cell": true, "toc_position": { "height": "40px", "left": "1560px", "right": "20px", "top": "133px", "width": "239px" }, "toc_section_display": "none", "toc_window_display": true, "widenNotebook": false }, "toc_position": { "height": "1081px", "left": "0px", "right": "1948px", "top": "107px", "width": "186px" } }, "nbformat": 4, "nbformat_minor": 1 }