chapter9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><h1>第九章 分类数据</h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、cat对象\n",
    "### 1. cat对象的属性\n",
    "在`pandas`中提供了`category`类型，使用户能够处理分类类型的变量，将一个普通序列转换成分类变量可以使用`astype`方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- cat对象的定义的属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     Freshman\n",
       "1     Freshman\n",
       "2       Senior\n",
       "3    Sophomore\n",
       "4    Sophomore\n",
       "Name: Grade, dtype: category\n",
       "Categories (4, object): ['Freshman', 'Junior', 'Senior', 'Sophomore']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "df = pd.read_csv('C:\\datawhale\\joyful-pandas-master\\data\\learn_pandas.csv', usecols = ['Grade', 'Name', 'Gender', 'Height', 'Weight'])\n",
    "s = df.Grade.astype('category')\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在一个分类类型的`Series`中定义了`cat`对象，它和上一章中介绍的`str`对象类似，定义了一些属性和方法来进行分类类别的操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.arrays.categorical.CategoricalAccessor object at 0x0000021306F01730>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.cat#cat对象"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于一个具体的分类，有两个组成部分，其一为类别的本身，它以`Index`类型存储，其二为是否有序，它们都可以通过`cat`的属性被访问："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Freshman', 'Junior', 'Senior', 'Sophomore'], dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.cat.categories#查看属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.cat.ordered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外，每一个序列的类别会被赋予唯一的整数编号，它们的编号取决于`cat.categories`中的顺序，该属性可以通过`codes`访问："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    2\n",
       "3    3\n",
       "4    3\n",
       "dtype: int8"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.cat.codes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 类别的增加、删除和修改\n",
    "\n",
    "通过`cat`对象的`categories`属性能够完成对类别的查询，那么应该如何进行“增改查删”的其他三个操作呢？\n",
    "\n",
    "\n",
    "首先，对于类别的增加可以使用`add_categories`："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- add_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Freshman', 'Junior', 'Senior', 'Sophomore', 'Graduate'], dtype='object')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.cat.add_categories('Graduate') # 增加一个毕业生类别\n",
    "s.cat.categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "若要删除某一个类别可以使用`remove_categories`，同时所有原来序列中的该类会被设置为缺失。例如，删除大一的类别："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- remove_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Junior', 'Senior', 'Sophomore', 'Graduate'], dtype='object')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.cat.remove_categories('Freshman')\n",
    "s.cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          NaN\n",
       "1          NaN\n",
       "2       Senior\n",
       "3    Sophomore\n",
       "4    Sophomore\n",
       "Name: Grade, dtype: category\n",
       "Categories (4, object): ['Junior', 'Senior', 'Sophomore', 'Graduate']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外可以使用`set_categories`直接设置序列的新类别，原来的类别中如果存在元素不属于新类别，那么会被设置为缺失。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- set_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sophomore', 'PhD'], dtype='object')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.cat.set_categories(['Sophomore','PhD']) # 新类别为大二学生和博士\n",
    "s.cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          NaN\n",
       "1          NaN\n",
       "2          NaN\n",
       "3    Sophomore\n",
       "4    Sophomore\n",
       "Name: Grade, dtype: category\n",
       "Categories (2, object): ['Sophomore', 'PhD']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果想要删除未出现在序列中的类别，可以使用`remove_unused_categories`来实现："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- remove_unused_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sophomore'], dtype='object')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.cat.remove_unused_categories() # 移除了未出现的类别PHD\n",
    "s.cat.categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，“增改查删”中还剩下修改的操作，这可以通过`rename_categories`方法完成，同时需要注意的是，这个方法会对原序列的对应值也进行相应修改。例如，现在把`Sophomore`改成中文的`本科二年级学生`："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- rename_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        NaN\n",
       "1        NaN\n",
       "2        NaN\n",
       "3    本科二年级学生\n",
       "4    本科二年级学生\n",
       "Name: Grade, dtype: category\n",
       "Categories (1, object): ['本科二年级学生']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.cat.rename_categories({'Sophomore':'本科二年级学生'})#索引的方式\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、有序分类\n",
    "### 1. 序的建立\n",
    "\n",
    "有序类别和无序类别可以通过`as_unordered`和`reorder_categories`互相转化，需要注意的是后者传入的参数必须是由当前序列的无需类别构成的列表，不能够增加新的类别，也不能缺少原来的类别，并且必须指定参数`ordered=True`，否则方法无效。例如，对年级高低进行相对大小的类别划分，然后再恢复无序状态："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 建立分类的顺序reorder与unordered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     Freshman\n",
       "1     Freshman\n",
       "2       Senior\n",
       "3    Sophomore\n",
       "4    Sophomore\n",
       "Name: Grade, dtype: category\n",
       "Categories (4, object): ['Freshman' < 'Sophomore' < 'Junior' < 'Senior']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = df.Grade.astype('category')\n",
    "s = s.cat.reorder_categories(['Freshman', 'Sophomore', 'Junior', 'Senior'],ordered=True)\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        NaN\n",
       "1        NaN\n",
       "2        NaN\n",
       "3    本科二年级学生\n",
       "4    本科二年级学生\n",
       "Name: Grade, dtype: category\n",
       "Categories (1, object): ['本科二年级学生']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.cat.as_unordered().head()#恢复"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 【NOTE】类别不得直接修改\n",
    "如果不想指定`ordered=True`参数，那么可以先用`s.cat.as_ordered()`转化为有序类别，再利用`reorder_categories`进行具体的相对大小调整。\n",
    "#### 【END】\n",
    "### 2. 排序和比较\n",
    "\n",
    "在第二章中，曾提到了字符串和数值类型序列的排序，此时就要说明分类变量的排序：只需把列的类型修改为`category`后，再赋予相应的大小关系，就能正常地使用`sort_index`和`sort_values`。例如，对年级进行排序："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- sort_values对排序好的分类进行排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Grade</th>\n",
       "      <th>Name</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Height</th>\n",
       "      <th>Weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Freshman</td>\n",
       "      <td>Gaopeng Yang</td>\n",
       "      <td>Female</td>\n",
       "      <td>158.9</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>Freshman</td>\n",
       "      <td>Qiang Shi</td>\n",
       "      <td>Female</td>\n",
       "      <td>164.5</td>\n",
       "      <td>52.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>Freshman</td>\n",
       "      <td>Changmei Feng</td>\n",
       "      <td>Female</td>\n",
       "      <td>163.8</td>\n",
       "      <td>56.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>Freshman</td>\n",
       "      <td>Xiaopeng Han</td>\n",
       "      <td>Female</td>\n",
       "      <td>164.1</td>\n",
       "      <td>53.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>Freshman</td>\n",
       "      <td>Yanli Zhang</td>\n",
       "      <td>Female</td>\n",
       "      <td>165.1</td>\n",
       "      <td>52.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Grade           Name  Gender  Height  Weight\n",
       "0    Freshman   Gaopeng Yang  Female   158.9    46.0\n",
       "105  Freshman      Qiang Shi  Female   164.5    52.0\n",
       "96   Freshman  Changmei Feng  Female   163.8    56.0\n",
       "88   Freshman   Xiaopeng Han  Female   164.1    53.0\n",
       "81   Freshman    Yanli Zhang  Female   165.1    52.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Grade = df.Grade.astype('category')\n",
    "df.Grade = df.Grade.cat.reorder_categories(['Freshman', 'Sophomore', 'Junior', 'Senior'],ordered=True)\n",
    "df.sort_values('Grade').head() # 值排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Height</th>\n",
       "      <th>Weight</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grade</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Freshman</th>\n",
       "      <td>Gaopeng Yang</td>\n",
       "      <td>Female</td>\n",
       "      <td>158.9</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Freshman</th>\n",
       "      <td>Qiang Shi</td>\n",
       "      <td>Female</td>\n",
       "      <td>164.5</td>\n",
       "      <td>52.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Freshman</th>\n",
       "      <td>Changmei Feng</td>\n",
       "      <td>Female</td>\n",
       "      <td>163.8</td>\n",
       "      <td>56.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Freshman</th>\n",
       "      <td>Xiaopeng Han</td>\n",
       "      <td>Female</td>\n",
       "      <td>164.1</td>\n",
       "      <td>53.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Freshman</th>\n",
       "      <td>Yanli Zhang</td>\n",
       "      <td>Female</td>\n",
       "      <td>165.1</td>\n",
       "      <td>52.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Name  Gender  Height  Weight\n",
       "Grade                                          \n",
       "Freshman   Gaopeng Yang  Female   158.9    46.0\n",
       "Freshman      Qiang Shi  Female   164.5    52.0\n",
       "Freshman  Changmei Feng  Female   163.8    56.0\n",
       "Freshman   Xiaopeng Han  Female   164.1    53.0\n",
       "Freshman    Yanli Zhang  Female   165.1    52.0"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('Grade').sort_index().head() # 先将分类设置成索引再对其进行索引排序"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于序的建立，因此就可以进行比较操作。分类变量的比较操作分为两类，第一种是`==`或`!=`关系的比较，比较的对象可以是标量或者同长度的`Series`（或`list`），第二种是`>,>=,<,<=`四类大小关系的比较，比较的对象和第一种类似，但是所有参与比较的元素必须属于原序列的`categories`，同时要和原序列具有相同的索引。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 排序后可以比较"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1    False\n",
       "2    False\n",
       "3     True\n",
       "4     True\n",
       "Name: Grade, dtype: bool"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res1 = df.Grade == 'Sophomore'\n",
    "res1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 5)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1    False\n",
       "2    False\n",
       "3    False\n",
       "4    False\n",
       "Name: Grade, dtype: bool"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res2 = df.Grade == ['PhD']*df.shape[0]#两百个phd\n",
    "res2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    True\n",
       "1    True\n",
       "2    True\n",
       "3    True\n",
       "4    True\n",
       "Name: Grade, dtype: bool"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res3 = df.Grade <= 'Sophomore'\n",
    "res3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2     True\n",
       "3    False\n",
       "4    False\n",
       "Name: Grade, dtype: bool"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res4 = df.Grade <= df.Grade.sample(frac=1).reset_index(drop=True) # 打乱后比较\n",
    "res4.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、区间类别\n",
    "\n",
    "### 1. 利用cut和qcut进行区间构造\n",
    "\n",
    "区间是一种特殊的类别，在实际数据分析中，区间序列往往是通过`cut`和`qcut`方法进行构造的，这两个函数能够把原序列的数值特征进行装箱，即用区间位置来代替原来的具体数值。\n",
    "\n",
    "首先介绍`cut`的常见用法：\n",
    "\n",
    "其中，最重要的参数是`bin`，如果传入整数`n`，则代表把整个传入数组的按照最大和最小值等间距地分为`n`段。由于区间默认是左开右闭，需要进行调整把最小值包含进去，在`pandas`中的解决方案是在值最小的区间左端点再减去`0.001*(max-min)`，因此如果对序列`[1,2]`划分为2个箱子时，第一个箱子的范围`(0.999,1.5]`，第二个箱子的范围是`(1.5,2]`。如果需要指定左闭右开时，需要把`right`参数设置为`False`，相应的区间调整方法是在值最大的区间右端点再加上`0.001*(max-min)`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- pd.cut设置bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    (0.999, 1.5]\n",
       "1      (1.5, 2.0]\n",
       "dtype: category\n",
       "Categories (2, interval[float64]): [(0.999, 1.5] < (1.5, 2.0]]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([1,2])\n",
    "pd.cut(s, bins=2)#划分成两截，右闭左开"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      [1.0, 1.5)\n",
       "1    [1.5, 2.001)\n",
       "dtype: category\n",
       "Categories (2, interval[float64]): [[1.0, 1.5) < [1.5, 2.001)]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.cut(s, bins=2, right=False)#左闭右开"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`bins`的另一个常见用法是指定区间分割点的列表（使用`np.infty`可以表示无穷大）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    (-inf, 1.2]\n",
       "1     (1.8, 2.2]\n",
       "dtype: category\n",
       "Categories (4, interval[float64]): [(-inf, 1.2] < (1.2, 1.8] < (1.8, 2.2] < (2.2, inf]]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.cut(s, bins=[-np.infty, 1.2, 1.8, 2.2, np.infty])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外两个常用参数为`labels`和`retbins`，分别代表了区间的名字和是否返回分割点（默认不返回）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    small\n",
       "1      big\n",
       "dtype: category\n",
       "Categories (2, object): ['small' < 'big']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([1,2])\n",
    "res = pd.cut(s, bins=2, labels=['small', 'big'], retbins=True)\n",
    "res[0]#返回标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.999, 1.5  , 2.   ])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res[1] # 该元素为返回的分割点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从用法上来说，`qcut`和`cut`几乎没有差别，只是把`bins`参数变成的`q`参数，`qcut`中的`q`是指`quantile`。这里的`q`为整数`n`时，指按照`n`等分位数把数据分箱，还可以传入浮点列表指代相应的分位数分割点。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- pd.qcut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    (33.999, 48.0]\n",
       "1      (55.0, 89.0]\n",
       "2      (55.0, 89.0]\n",
       "3    (33.999, 48.0]\n",
       "4      (55.0, 89.0]\n",
       "Name: Weight, dtype: category\n",
       "Categories (3, interval[float64]): [(33.999, 48.0] < (48.0, 55.0] < (55.0, 89.0]]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = df.Weight\n",
    "pd.qcut(s, q=3).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      (44.0, 69.4]\n",
       "1      (69.4, 89.0]\n",
       "2      (69.4, 89.0]\n",
       "3    (33.999, 44.0]\n",
       "4      (69.4, 89.0]\n",
       "Name: Weight, dtype: category\n",
       "Categories (3, interval[float64]): [(33.999, 44.0] < (44.0, 69.4] < (69.4, 89.0]]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.qcut(s, q=[0,0.2,0.8,1]).head()#q表示分位数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 一般区间的构造\n",
    "\n",
    "对于某一个具体的区间而言，其具备三个要素，即左端点、右端点和端点的开闭状态，其中开闭状态可以指定`right, left, both, neither`中的一类："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Interval(0, 1, closed='right')"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_interval = pd.Interval(0, 1, 'right')#pd.Interval设置区间，右边闭\n",
    "my_interval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其属性包含了`mid, length, right, left, closed,`，分别表示中点、长度、右端点、左端点和开闭状态。\n",
    "\n",
    "使用`in`可以判断元素是否属于区间："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.5 in my_interval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0 in my_interval#0不在   （0，1】"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用`overlaps`可以判断两个区间是否有交集："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- overlaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_interval_2 = pd.Interval(0.5, 1.5, 'left')\n",
    "my_interval.overlaps(my_interval_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般而言，`pd.IntervalIndex`对象有四类方法生成，分别是`from_breaks, from_arrays, from_tuples, interval_range`，它们分别应用于不同的情况：\n",
    "\n",
    "`from_breaks`的功能类似于`cut`或`qcut`函数，只不过后两个是通过计算得到的风格点，而前者是直接传入自定义的分割点："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 生成区间序列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([[1, 3], [3, 6], [6, 10]],\n",
       "              closed='both',\n",
       "              dtype='interval[int64]')"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.IntervalIndex.from_breaks([1,3,6,10], closed='both')#设定节点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`from_arrays`是分别传入左端点和右端点的列表，适用于有交集并且知道起点和终点的情况："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(1, 5), (3, 4), (6, 9), (10, 11)],\n",
       "              closed='neither',\n",
       "              dtype='interval[int64]')"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.IntervalIndex.from_arrays(left = [1,3,6,10], right = [5,4,9,11], closed = 'neither')#设定左右端点的列表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`from_tuples`传入的是起点和终点元组构成的列表："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(1, 5), (3, 4), (6, 9), (10, 11)],\n",
       "              closed='neither',\n",
       "              dtype='interval[int64]')"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.IntervalIndex.from_tuples([(1,5),(3,4),(6,9),(10,11)], closed='neither')#设定每个区间的列表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个等差的区间序列由起点、终点、区间个数和区间长度决定，其中三个量确定的情况下，剩下一个量就确定了，`interval_range`中的`start, end, periods, freq`参数就对应了这四个量，从而就能构造出相应的区间："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(1.0, 1.5], (1.5, 2.0], (2.0, 2.5], (2.5, 3.0], (3.0, 3.5], (3.5, 4.0], (4.0, 4.5], (4.5, 5.0]],\n",
       "              closed='right',\n",
       "              dtype='interval[float64]')"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.interval_range(start=1,end=5,periods=8)#等差序列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(1.0, 1.5], (1.5, 2.0], (2.0, 2.5], (2.5, 3.0], (3.0, 3.5], (3.5, 4.0], (4.0, 4.5], (4.5, 5.0]],\n",
       "              closed='right',\n",
       "              dtype='interval[float64]')"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.interval_range(end=5,periods=8,freq=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 【练一练】\n",
    "无论是`interval_range`还是下一章时间序列中的`date_range`都是给定了等差序列中四要素中的三个，从而确定整个序列。请回顾等差数列中的首项、末项、项数和公差的联系，写出`interval_range`中四个参数之间的恒等关系。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "end-start=periods*freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 【end】"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除此之外，如果直接使用`pd.IntervalIndex([...], closed=...)`，把`Interval`类型的列表组成传入其中转为区间索引，那么所有的区间会被强制转为指定的`closed`类型，因为`pd.IntervalIndex`只允许存放同一种开闭区间的`Interval`对象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([[0.0, 1.0), [0.5, 1.5)],\n",
       "              closed='left',\n",
       "              dtype='interval[float64]')"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.IntervalIndex([my_interval, my_interval_2], closed='left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 区间的属性与方法\n",
    "\n",
    "`IntervalIndex`上也定义了一些有用的属性和方法。同时，如果想要具体利用`cut`或者`qcut`的结果进行分析，那么需要先将其转为该种索引类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    46.0\n",
       "1    70.0\n",
       "2    89.0\n",
       "3    41.0\n",
       "4    74.0\n",
       "Name: Weight, dtype: float64"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    (33.945, 52.333]\n",
       "1    (52.333, 70.667]\n",
       "2      (70.667, 89.0]\n",
       "3    (33.945, 52.333]\n",
       "4      (70.667, 89.0]\n",
       "Name: Weight, dtype: category\n",
       "Categories (3, interval[float64]): [(33.945, 52.333] < (52.333, 70.667] < (70.667, 89.0]]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.cut(s, 3).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(33.945, 52.333], (52.333, 70.667], (70.667, 89.0]],\n",
       "              closed='right',\n",
       "              name='Weight',\n",
       "              dtype='interval[float64]')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_interval = pd.IntervalIndex(pd.cut(s, 3))\n",
    "id_interval[:3]#前三个"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与单个`Interval`类型相似，`IntervalIndex`有若干常用属性：`left, right, mid, length`，分别表示左右端点、两端点均值和区间长度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([(33.945, 52.333], (52.333, 70.667], (70.667, 89.0], (33.945, 52.333], (70.667, 89.0]],\n",
       "              closed='right',\n",
       "              name='Weight',\n",
       "              dtype='interval[float64]')"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo = id_interval[:5] # 选出前5个展示\n",
    "id_demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 区间序列属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Float64Index([33.945, 52.333, 70.667, 33.945, 70.667], dtype='float64')"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo.left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Float64Index([52.333, 70.667, 89.0, 52.333, 89.0], dtype='float64')"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo.right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Float64Index([43.138999999999996, 61.5, 79.8335, 43.138999999999996, 79.8335], dtype='float64')"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo.mid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Float64Index([18.387999999999998, 18.334000000000003, 18.333,\n",
       "              18.387999999999998, 18.333],\n",
       "             dtype='float64')"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo.length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`IntervalIndex`还有两个常用方法，包括`contains`和`overlaps`，分别指逐个判断每个区间是否包含某元素，以及是否和一个`pd.Interval`对象有交集。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 区间序列方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, False, False])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo.contains(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True, False,  True, False])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_demo.overlaps(pd.Interval(40,60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 四、练习\n",
    "### Ex1：统计未出现的类别\n",
    "\n",
    "在第五章中介绍了`crosstab`函数，在默认参数下它能够对两个列的组合出现的频数进行统计汇总："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>B</th>\n",
       "      <th>cat</th>\n",
       "      <th>dog</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "B  cat  dog\n",
       "A          \n",
       "a    2    0\n",
       "b    1    0\n",
       "c    0    1"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'A':['a','b','c','a'], 'B':['cat','cat','dog','cat']})\n",
    "pd.crosstab(df.A, df.B)#透视表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但事实上有些列存储的是分类变量，列中并不一定包含所有的类别，此时如果想要对这些未出现的类别在`crosstab`结果中也进行汇总，则可以指定`dropna`参数为`False`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>B</th>\n",
       "      <th>cat</th>\n",
       "      <th>dog</th>\n",
       "      <th>sheep</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "B  cat  dog  sheep\n",
       "A                 \n",
       "a    2    0      0\n",
       "b    1    0      0\n",
       "c    0    1      0"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.B = df.B.astype('category').cat.add_categories('sheep')\n",
    "pd.crosstab(df.A, df.B,dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请实现一个带有`dropna`参数的`my_crosstab`函数来完成上面的功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>B</th>\n",
       "      <th>cat</th>\n",
       "      <th>dog</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "B  cat  dog\n",
       "A          \n",
       "a    2    0\n",
       "b    1    0\n",
       "c    0    1"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def my_crosstab(s1,s2,dropna=True):\n",
    "    idx1 = (s1.cat.categories if s1.dtype.name == 'category' and not dropna else s1.unique())#dropna为false且s1为分类属性取出s1的分类，否则取出独一无二的s1\n",
    "    idx2 = (s2.cat.categories if s2.dtype.name == 'category' and not dropna else s2.unique())#dropna为false且s2为分类属性\n",
    "    res = pd.DataFrame(np.zeros((idx1.shape[0], idx2.shape[0])),index=idx1, columns=idx2)#索引为1，列为2的一个空表\n",
    "    for i, j in zip(s1, s2):#横坐标为s1，纵坐标为s2时表格中数据加一\n",
    "        res.at[i, j] += 1\n",
    "    res = res.rename_axis(index=s1.name, columns=s2.name).astype('int')#类型转化成整数型\n",
    "    return res\n",
    "\n",
    "df = pd.DataFrame({'A':['a','b','c','a'], 'B':['cat','cat','dog','cat']})\n",
    "df.B = df.B.astype('category').cat.add_categories('sheep')#将df.B设置成分类属性并添加一个不属于其中的元素\n",
    "my_crosstab(df.A, df.B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>B</th>\n",
       "      <th>cat</th>\n",
       "      <th>dog</th>\n",
       "      <th>sheep</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "B  cat  dog  sheep\n",
       "A                 \n",
       "a    2    0      0\n",
       "b    1    0      0\n",
       "c    0    1      0"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_crosstab(df.A, df.B,dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ex2：钻石数据集\n",
    "\n",
    "现有一份关于钻石的数据集，其中`carat, cut, clarity, price`分别表示克拉重量、切割质量、纯净度和价格，样例如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>clarity</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>SI2</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>SI1</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>VS1</td>\n",
       "      <td>327</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat      cut clarity  price\n",
       "0   0.23    Ideal     SI2    326\n",
       "1   0.21  Premium     SI1    326\n",
       "2   0.23     Good     VS1    327"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('C:\\datawhale\\joyful-pandas-master\\data\\diamonds.csv') \n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 分别对`df.cut`在`object`类型和`category`类型下使用`nunique`函数，并比较它们的性能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.36 ms ± 829 µs per loop (mean ± std. dev. of 7 runs, 30 loops each)\n"
     ]
    }
   ],
   "source": [
    "s_obj, s_cat = df.cut, df.cut.astype('category')\n",
    "%timeit -n 30 s_obj.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.94 ms ± 851 µs per loop (mean ± std. dev. of 7 runs, 30 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit -n 30 s_cat.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 钻石的切割质量可以分为五个等级，由次到好分别是`Fair, Good, Very Good, Premium, Ideal`，纯净度有八个等级，由次到好分别是`I1, SI2, SI1, VS2, VS1, VVS2, VVS1, IF`，请对切割质量按照**由好到次**的顺序排序，相同切割质量的钻石，按照纯净度进行**由次到好**的排序。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.cut = df.cut.astype('category').cat.reorder_categories(['Fair', 'Good', 'Very Good', 'Premium', 'Ideal'],ordered=True)\n",
    "df.clarity = df.clarity.astype('category').cat.reorder_categories(['I1', 'SI2', 'SI1', 'VS2', 'VS1', 'VVS2', 'VVS1', 'IF'],ordered=True)\n",
    "#设定号切割质量与纯净度的分类属性及排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>clarity</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>0.96</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>I1</td>\n",
       "      <td>2801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>535</th>\n",
       "      <td>0.96</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>I1</td>\n",
       "      <td>2826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>551</th>\n",
       "      <td>0.97</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>I1</td>\n",
       "      <td>2830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>653</th>\n",
       "      <td>1.01</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>I1</td>\n",
       "      <td>2844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>718</th>\n",
       "      <td>0.97</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>I1</td>\n",
       "      <td>2856</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     carat    cut clarity  price\n",
       "315   0.96  Ideal      I1   2801\n",
       "535   0.96  Ideal      I1   2826\n",
       "551   0.97  Ideal      I1   2830\n",
       "653   1.01  Ideal      I1   2844\n",
       "718   0.97  Ideal      I1   2856"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = df.sort_values(['cut', 'clarity'], ascending=[False, True])#sort_values排序\n",
    "res.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 分别采用两种不同的方法，把`cut, clarity`这两列按照**由好到次**的顺序，映射到从0到n-1的整数，其中n表示类别的个数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 方法一:cat.codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>clarity</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>0</td>\n",
       "      <td>SI2</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>1</td>\n",
       "      <td>SI1</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>3</td>\n",
       "      <td>VS1</td>\n",
       "      <td>327</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat  cut clarity  price\n",
       "0   0.23    0     SI2    326\n",
       "1   0.21    1     SI1    326\n",
       "2   0.23    3     VS1    327"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.cut = df.cut.cat.reorder_categories(df.cut.cat.categories[::-1])\n",
    "df.clarity = df.clarity.cat.reorder_categories(df.clarity.cat.categories[::-1])\n",
    "df.cut = df.cut.cat.codes \n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 方法二：replace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>clarity</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>327</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat  cut  clarity  price\n",
       "0   0.23    0        6    326\n",
       "1   0.21    1        5    326\n",
       "2   0.23    3        3    327"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clarity_cat = df.clarity.cat.categories\n",
    "df.clarity = df.clarity.replace(dict(zip(clarity_cat, np.arange(len(clarity_cat)))))\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 对每克拉的价格按照分别按照分位数（q=\\[0.2, 0.4, 0.6, 0.8\\]）与\\[1000, 3500, 5500, 18000\\]割点进行分箱得到五个类别`Very Low, Low, Mid, High, Very High`，并把按这两种分箱方法得到的`category`序列依次添加到原表中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>clarity</th>\n",
       "      <th>price</th>\n",
       "      <th>avg_cut</th>\n",
       "      <th>avg_qcut</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>326</td>\n",
       "      <td>Low</td>\n",
       "      <td>Very Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>326</td>\n",
       "      <td>Low</td>\n",
       "      <td>Very Low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>327</td>\n",
       "      <td>Low</td>\n",
       "      <td>Very Low</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat  cut  clarity  price avg_cut  avg_qcut\n",
       "0   0.23    0        6    326     Low  Very Low\n",
       "1   0.21    1        5    326     Low  Very Low\n",
       "2   0.23    3        3    327     Low  Very Low"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = [0, 0.2, 0.4, 0.6, 0.8, 1]\n",
    "point = [-np.infty, 1000, 3500, 5500, 18000, np.infty]\n",
    "avg = df.price / df.carat#单位克拉价格\n",
    "df['avg_cut'] = pd.cut(avg, bins=point, labels=['Very Low', 'Low', 'Mid', 'High', 'Very High'])\n",
    "df['avg_qcut'] = pd.qcut(avg, q=q, labels=['Very Low', 'Low', 'Mid', 'High', 'Very High'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. 第4问中按照整数分箱得到的序列中，是否出现了所有的类别？如果存在没有出现的类别请把该类别删除。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Low', 'Mid', 'High']\n",
       "Categories (3, object): ['Low' < 'Mid' < 'High']"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.avg_cut.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Very Low', 'Low', 'Mid', 'High', 'Very High'], dtype='object')"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.avg_cut.cat.categories#没有出现所有分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.avg_cut = df.avg_cut.cat.remove_categories(['Very Low', 'Very High'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. 对第4问中按照分位数分箱得到的序列，求每个样本对应所在区间的左右端点值和长度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2295.0\n",
       "1    2295.0\n",
       "2    2295.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interval_avg = pd.IntervalIndex(pd.qcut(avg, q=q))#按分位数分箱的序列\n",
    "interval_avg.right.to_series().reset_index(drop=True).head(3)#右端点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1051.162\n",
       "1    1051.162\n",
       "2    1051.162\n",
       "dtype: float64"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interval_avg.left.to_series().reset_index(drop=True).head(3)#左端点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1243.838\n",
       "1    1243.838\n",
       "2    1243.838\n",
       "dtype: float64"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interval_avg.length.to_series().reset_index(drop=True).head(3)#长度"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}