python-sdk/tutorials/nuscenes_lidarseg_panoptic_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# nuScenes lidarseg and panoptic tutorial\n",
    "\n",
    "Welcome to the nuScenes lidarseg and panoptic tutorial. The lidarseg and panoptic share quite many functions in the tutorial, so we put them into single tutorial. But you can opt to set up either the lidarseg or the panoptic dataset, and only run the portion for specific task.\n",
    "\n",
    "This demo assumes that nuScenes is installed at `/data/sets/nuscenes`. The mini version (i.e. v1.0-mini) of the full dataset will be used for this demo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "To install the nuScenes-lidarseg and/or Panoptic nuScenes expansion, download the dataset from https://www.nuscenes.org/download. Unpack the compressed file(s) into `/data/sets/nuscenes` and your folder structure should end up looking like this:\n",
    "```\n",
    "└── nuscenes  \n",
    "    ├── Usual nuscenes folders (i.e. samples, sweep)\n",
    "    │\n",
    "    ├── lidarseg\n",
    "    │   └── v1.0-{mini, test, trainval} <- Contains the .bin files; a .bin file \n",
    "    │                                      contains the labels of the points in a \n",
    "    │                                      point cloud (note that v1.0-test does not \n",
    "    │                                      have any .bin files associated with it)\n",
    "    │\n",
    "    ├── panoptic\n",
    "    │   └── v1.0-{mini, test, trainval} <- Contains the *_panoptic.npz files; a .npz file \n",
    "    │                                      contains the panoptic labels of the points in a \n",
    "    │                                      point cloud (note that v1.0-test does not \n",
    "    │                                      have any .npz files associated with it) \n",
    "    └── v1.0-{mini, test, trainval}\n",
    "        ├── Usual files (e.g. attribute.json, calibrated_sensor.json etc.)\n",
    "        ├── lidarseg.json  <- contains the mapping of each .bin file to the token\n",
    "        ├── panoptic.json  <- contains the mapping of each .npz file to the token       \n",
    "        └── category.json  <- contains the categories of the labels (note that the \n",
    "                              category.json from nuScenes v1.0 is overwritten)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Colab (optional) \n",
    "\n",
    "<br>\n",
    "<a href=\"https://colab.research.google.com/github/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/nuscenes_lidarseg_panoptic_tutorial.ipynb\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/ align=\"left\">\n",
    "</a>\n",
    "<br>\n",
    "\n",
    "If you are running this notebook in Google Colab, you can uncomment the cell below and run it; everything will be set up nicely for you. Otherwise, go to [**Setup**](#Setup) to manually set up everything."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download and setup nuScenes-devkit for nuScenes-lidarseg dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !mkdir -p /data/sets/nuscenes  # Make the directory to store the nuScenes dataset in.\n",
    "\n",
    "# !wget https://www.nuscenes.org/data/v1.0-mini.tgz  # Download the nuScenes mini split.\n",
    "# !wget https://www.nuscenes.org/data/nuScenes-lidarseg-mini-v1.0.tar.bz2  # Download the nuScenes-lidarseg mini split.\n",
    "\n",
    "# !tar -xf v1.0-mini.tgz -C /data/sets/nuscenes  # Uncompress the nuScenes mini split.\n",
    "# !tar -xf nuScenes-lidarseg-mini-v1.0.tar.bz2 -C /data/sets/nuscenes   # Uncompress the nuScenes-lidarseg mini split.\n",
    "\n",
    "# !pip install nuscenes-devkit &> /dev/null  # Install nuScenes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download and setup Panoptic nuScenes dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !wget https://www.nuscenes.org/data/v1.0-mini.tgz  # Download the nuScenes mini split.\n",
    "# !wget https://www.nuscenes.org/data/nuScenes-panoptic-v1.0-mini.tar.gz  # Download the Panoptic nuScenes mini split.\n",
    "\n",
    "# !tar -xf v1.0-mini.tgz -C /data/sets/nuscenes  # Uncompress the nuScenes mini split.\n",
    "# !tar -xf nuScenes-panoptic-v1.0-mini.tar.gz -C /data/sets/nuscenes   # Uncompress the Panoptic nuScenes mini split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "Let's start by importing the necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from nuscenes import NuScenes\n",
    "\n",
    "nusc = NuScenes(version='v1.0-mini', dataroot='/data/sets/nuscenes', verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, you do not need any extra libraries to use nuScenes-lidarseg and Panoptic nuScenes. The original nuScenes devkit which you are familiar with has been extended so that you can use it seamlessly with nuScenes-lidarseg and Panoptic nuScenes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Point statistics of lidarseg/panoptic dataset for the v1.0-mini split\n",
    "Let's get a quick feel of the lidarseg dataset by looking at what classes are in it and the number of points belonging to each class. The classes will be sorted in ascending order based on the number of points (since `sort_by='count'` below); you can also sort the classes by class name or class index by setting `sort_by='name'` or `sort_by='index'` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuscenes-lidarseg\n",
    "nusc.list_lidarseg_categories(sort_by='count')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `list_lidarseg_categories`, you can get the index which each class name belongs to by looking at the leftmost column. You can also get a mapping of the indices to the class names from the `lidarseg_idx2name_mapping` attribute of the NuScenes class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.lidarseg_idx2name_mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conversely, you can get the mapping of the class names to the indices from the `lidarseg_name2idx_mapping` attribute of the NuScenes class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.lidarseg_name2idx_mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Panoptic nuScenes, it shares the same member variables `lidarseg_idx2name_mapping` and `lidarseg_names2idx_mapping` with nuScenes-lidarseg. Similarly, you can check the number of points for each semantic category from the Panoptic nuScenes dataset. The only thing to do is add `gt_from='panoptic'` argument. By default, `gt_from='lidarseg'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Panoptic nuScenes\n",
    "nusc.list_lidarseg_categories(sort_by='count', gt_from='panoptic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might have noticed the point numbers for certain categories vary slightly between lidarseg and panoptic dataset. The reason is the overlapping points between instances are set to noise (category 0) in Panoptic nuScenes. You can see the increased number of points for `noise` category in Panoptic nuScenes, and the total point number remains the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instance statistics of panoptic dataset  for the v1.0-mini split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instances statistics are specific to panoptic dataset. We provide `list_panoptic_instances()` function for this purpose. You can set the `sort_by` to one of `['count', 'index', 'name']`. The function will calculate the number of instances per frame, total number of instances (unique object ID) and instance states (one instance could have more than one states, a.k.a, a track). Also it calculates the per-category statistics, including the mean and standard deviation for number of frames an instance spans, and mean and standard deviation for number of points per instance.\n",
    "\n",
    "Note only thing categories have instances. The point statistics could refer to the point statistics section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_panoptic_instances(sort_by='count')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pick a sample token\n",
    "Let's pick a sample to use for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_sample = nusc.sample[87]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get statistics of a lidarseg/panoptic sample token\n",
    "Now let's take a look at what classes are present in the pointcloud of this particular sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuscenes-lidarseg\n",
    "nusc.get_sample_lidarseg_stats(my_sample['token'], sort_by='count')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By doing `sort_by='count'`, the classes and their respective frequency counts are printed in ascending order; you can also do `sort_by='name'` and `sort_by='index'` here as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can use the same function to get the category frequency counts using the panoptic dataset by adding `gt_from='panoptic'`. As mentioned in `list_lidarseg_categories()`, the point count might be slightly different to lidarseg, due to the overlapping points of multiple instances are set to `noise` in Panoptic nuScenes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Panoptic nuScenes\n",
    "nusc.get_sample_lidarseg_stats(my_sample['token'], sort_by='count', gt_from='panoptic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render the lidarseg labels in the bird's eye view of a pointcloud\n",
    "In the original nuScenes devkit, you would pass a sample data token into ```render_sample_data``` to render a bird's eye view of the pointcloud. However, the points would be colored according to the distance from the ego vehicle. Now with the extended nuScenes devkit, all you need to do is set ```show_lidarseg=True``` to visualize the class labels of the pointcloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_data_token = my_sample['data']['LIDAR_TOP']\n",
    "nusc.render_sample_data(sample_data_token,\n",
    "                        with_anns=False,\n",
    "                        show_lidarseg=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But what if you wanted to focus on only certain classes? Given the statistics of the pointcloud printed out previously, let's say you are only interested in trucks and trailers. You could see the class indices belonging to those classes from the statistics and then pass an array of those indices into ```filter_lidarseg_labels``` like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_sample_data(sample_data_token,\n",
    "                        with_anns=False,\n",
    "                        show_lidarseg=True,\n",
    "                        filter_lidarseg_labels=[22, 23])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now only points in the pointcloud belonging to trucks and trailers are filtered out for your viewing pleasure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, you can display a legend which indicates the color for each class by using `show_lidarseg_legend`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_sample_data(sample_data_token,\n",
    "                        with_anns=False,\n",
    "                        show_lidarseg=True,\n",
    "                        show_lidarseg_legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render the panoptic labels in the bird's eye view of a pointcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to lidarseg, the same function is used to render the panoptic labels. The argument difference is `show_panoptic=True`. By default, both `show_lidarseg` and `show_panoptic` are set to `False`. If both are set to `True`, i.e. `show_lidarseg=True, show_panoptic=True`, lidarseg will have the priority to render."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_data_token = my_sample['data']['LIDAR_TOP']\n",
    "nusc.render_sample_data(sample_data_token,\n",
    "                        with_anns=False,\n",
    "                        show_lidarseg=False,\n",
    "                        show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see different vehicle instances from the same category will be displayed with unique colors. Similarly, you can play with the `filter_lidarseg_labels` and `show_lidarseg_legend=True` to show panoptic labels for certain thing and stuff categories, and the category legends. Note these 2 arguments are shared between lidarseg and panoptic datasets as well. Only legends of stuff categories will be displayed as the thing instances of same category have different colors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show trucks, trailers and drivable_surface\n",
    "nusc.render_sample_data(sample_data_token,\n",
    "                        with_anns=False,\n",
    "                        show_panoptic=True,\n",
    "                        filter_lidarseg_labels=[22, 23, 24])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show stuff category legends\n",
    "nusc.render_sample_data(sample_data_token,\n",
    "                        with_anns=False,\n",
    "                        show_lidarseg=False,\n",
    "                        show_lidarseg_legend=True,\n",
    "                        show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render lidarseg/panoptic labels in image\n",
    "If you wanted to superimpose the pointcloud into the corresponding image from a camera, you can use ```render_pointcloud_in_image``` like what you would do with the original nuScenes devkit, but set ```show_lidarseg=True``` (remember to set ```render_intensity=False```). Similar to ```render_sample_data```, you can filter to see only certain classes using ```filter_lidarseg_labels```. And you can use ```show_lidarseg_legend``` to display a legend in the rendering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuscenes-lidarseg\n",
    "nusc.render_pointcloud_in_image(my_sample['token'],\n",
    "                                pointsensor_channel='LIDAR_TOP',\n",
    "                                camera_channel='CAM_BACK',\n",
    "                                render_intensity=False,\n",
    "                                show_lidarseg=True,\n",
    "                                filter_lidarseg_labels=[22, 23, 24],\n",
    "                                show_lidarseg_legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this function supports `show_panoptic=True` mode, panoptic labels will be displayed rather than semantic labels. Only legends for stuff categories will be displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Panoptic nuScenes\n",
    "nusc.render_pointcloud_in_image(my_sample['token'],\n",
    "                                pointsensor_channel='LIDAR_TOP',\n",
    "                                camera_channel='CAM_BACK',\n",
    "                                render_intensity=False,\n",
    "                                show_lidarseg=False,\n",
    "                                filter_lidarseg_labels=[17,22, 23, 24],\n",
    "                                show_lidarseg_legend=True,\n",
    "                                show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render sample (i.e. lidar, radar and all cameras)\n",
    "Of course, like in the original nuScenes devkit, you can render all the sensors at once with ```render_sample```. In this extended nuScenes devkit, you can set ```show_lidarseg=True``` to see the lidarseg labels. Similar to the above methods, you can use ```filter_lidarseg_labels``` to display only the classes you wish to see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuscenes-lidarseg\n",
    "nusc.render_sample(my_sample['token'],\n",
    "                   show_lidarseg=True,\n",
    "                   filter_lidarseg_labels=[22, 23])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show panoptic labels with `render_sample`, set `show_panoptic=True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Panoptic nuScenes\n",
    "nusc.render_sample(my_sample['token'],\n",
    "                   show_lidarseg=False,\n",
    "                   filter_lidarseg_labels=[17, 23, 24],\n",
    "                   show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render a scene for a given camera sensor with lidarseg/panoptic labels\n",
    "You can also render an entire scene with the lidarseg labels for a camera of your choosing (the ```filter_lidarseg_labels``` argument can be used here as well)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's pick a scene first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_scene = nusc.scene[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then pass the scene token into ```render_scene_channel_lidarseg``` indicating that we are only interested in construction vehicles and man-made objects (here, we set `verbose=True` to produce a window which will allows us to see the frames as they are being random). \n",
    "\n",
    "In addition, you can use `dpi` (to adjust the size of the lidar points) and `imsize` (to adjust the size of the rendered image) to tune the aesthetics of the renderings to your liking.\n",
    "\n",
    "(Note: the following code is commented out as it crashes in Jupyter notebooks.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # nuscenes-lidarseg\n",
    "# import os\n",
    "# nusc.render_scene_channel_lidarseg(my_scene['token'], \n",
    "#                                    'CAM_BACK', \n",
    "#                                    filter_lidarseg_labels=[18, 28],\n",
    "#                                    verbose=True, \n",
    "#                                    dpi=100,\n",
    "#                                    imsize=(1280, 720))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function also works for panoptic labels, by adding `show_panoptic=True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # # Panoptic nuScenes\n",
    "# import os\n",
    "# nusc.render_scene_channel_lidarseg(my_scene['token'], \n",
    "#                                    'CAM_BACK', \n",
    "#                                    filter_lidarseg_labels=[18, 24, 28],\n",
    "#                                    verbose=True, \n",
    "#                                    dpi=100,\n",
    "#                                    imsize=(1280, 720),\n",
    "#                                    show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save the renderings, you can pass a path to a folder you want to save the images to via the ```out_folder``` argument, and either `video` or `image` to `render_mode`.\n",
    "\n",
    "(Note: the following code is commented out as it crashes in Jupyter notebooks.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # nuscenes-lidarseg\n",
    "# nusc.render_scene_channel_lidarseg(my_scene['token'],\n",
    "#                                    'CAM_BACK',\n",
    "#                                    filter_lidarseg_labels=[18, 28],\n",
    "#                                    verbose=True,\n",
    "#                                    dpi=100,\n",
    "#                                    imsize=(1280, 720),\n",
    "#                                    render_mode='video',\n",
    "#                                    out_folder=os.path.expanduser('~/Desktop/my_folder'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When `render_mode='image'`, only frames which contain points (after the filter has been applied) will be saved as images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also the same function can be used to render scene channel for panoptic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Panoptic nuScenes\n",
    "# nusc.render_scene_channel_lidarseg(my_scene['token'],\n",
    "#                                    'CAM_BACK',\n",
    "#                                    filter_lidarseg_labels=[18, 24, 28],\n",
    "#                                    verbose=True,\n",
    "#                                    dpi=100,\n",
    "#                                    imsize=(1280, 720),\n",
    "#                                    render_mode='video',\n",
    "#                                    out_folder=os.path.expanduser('~/Desktop/my_folder'),\n",
    "#                                    show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render a scene for all cameras with lidarseg/panoptic labels\n",
    "You can also render the entire scene for all cameras at once with the lidarseg labels as a video. Let's say in this case, we are interested in points belonging to driveable surfaces and cars.\n",
    "\n",
    "(Note: the following code is commented out as it crashes in Jupyter notebooks.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# # nuscenes-lidarseg\n",
    "# nusc.render_scene_lidarseg(my_scene['token'], \n",
    "#                            filter_lidarseg_labels=[17, 24],\n",
    "#                            verbose=True,\n",
    "#                            dpi=100,\n",
    "#                            out_path=os.path.expanduser('~/Desktop/my_rendered_scene.avi'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we can render a scene for panoptic labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Panoptic nuScenes\n",
    "# nusc.render_scene_lidarseg(my_scene['token'], \n",
    "#                            filter_lidarseg_labels=[17, 24],\n",
    "#                            verbose=True,\n",
    "#                            dpi=100,\n",
    "#                            out_path=os.path.expanduser('~/Desktop/my_rendered_scene.avi')\n",
    "#                            show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing LiDAR segmentation predictions\n",
    "In all the above functions, the labels of the LiDAR pointcloud which have been rendered are the ground truth. If you have trained a model to segment LiDAR pointclouds and have run it on the nuScenes-lidarseg dataset, you can visualize your model's predictions with nuScenes-lidarseg as well!\n",
    "\n",
    "Each of your .bin files should be a `numpy.uint8` array; as a tip, you can save your predictions as follows:\n",
    "```\n",
    "np.array(predictions).astype(np.uint8).tofile(bin_file_out)\n",
    "```\n",
    "- `predictions`: The predictions from your model (e.g. `[30, 5, 18, ..., 30]`)\n",
    "- `bin_file_out`: The path to write your .bin file to (e.g. `/some/folder/<lidar_sample_data_token>_lidarseg.bin`)\n",
    "\n",
    "Then you simply need to pass the path to the .bin file where your predictions for the given sample are to `lidarseg_preds_bin_path` for these functions:\n",
    "- `list_lidarseg_categories`\n",
    "- `render_sample_data`\n",
    "- `render_pointcloud_in_image`\n",
    "- `render_sample`                 \n",
    "\n",
    "For example, let's assume the predictions for `my_sample` is stored at `/data/sets/nuscenes/lidarseg/v1.0-mini` with the format `<lidar_sample_data_token>_lidarseg.bin`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "my_sample = nusc.sample[87]\n",
    "sample_data_token = my_sample['data']['LIDAR_TOP']\n",
    "my_predictions_bin_file = os.path.join('/data/sets/nuscenes/lidarseg/v1.0-mini', sample_data_token + '_lidarseg.bin')\n",
    "\n",
    "nusc.render_pointcloud_in_image(my_sample['token'],\n",
    "                                pointsensor_channel='LIDAR_TOP',\n",
    "                                camera_channel='CAM_BACK',\n",
    "                                render_intensity=False,\n",
    "                                show_lidarseg=True,\n",
    "                                filter_lidarseg_labels=[22, 23],\n",
    "                                show_lidarseg_legend=True,\n",
    "                                lidarseg_preds_bin_path=my_predictions_bin_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these functions that render an entire scene, you will need to pass the path to the folder which contains the .bin files for each sample in a scene to `lidarseg_preds_folder`:\n",
    "- `render_scene_channel_lidarseg`\n",
    "- `render_scene_lidarseg`\n",
    "\n",
    "Pay special attention that **each set of predictions in the folder _must_ be a `.bin` file and named as `<lidar_sample_data_token>_lidarseg.bin`**.\n",
    "\n",
    "(Note: the following code is commented out as it crashes in Jupyter notebooks.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# my_scene = nusc.scene[0]\n",
    "# my_folder_of_predictions = '/data/sets/nuscenes/lidarseg/v1.0-mini'\n",
    "\n",
    "# nusc.render_scene_channel_lidarseg(my_scene['token'], \n",
    "#                                    'CAM_BACK', \n",
    "#                                    filter_lidarseg_labels=[17, 24],\n",
    "#                                    verbose=True, \n",
    "#                                    imsize=(1280, 720),\n",
    "#                                    lidarseg_preds_folder=my_folder_of_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize LiDAR panoptic predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, panoptic prediction results could be rendered as well!\n",
    "Each of your .npz files should be a compressed `numpy.uint16` array; You can save your predictions as follows:\n",
    "```\n",
    "np.savez_compressed(npz_file_out, data=predictions.astype(np.uint16))\n",
    "```\n",
    "- `predictions`: The predictions from your model (e.g. `[1030, 15005, 180, ..., 3030]`, the `panoptic label = category_id * 1000 + instance_id`, the instance_id is unique, i.e. `[1, 2, 3, ..., 999]` within each scene).\n",
    "- `bin_file_out`: The path to write your .npz file to (e.g. `/some/folder/<lidar_sample_data_token>_panoptic.npz`). Note \n",
    "\n",
    "Then you simply need to pass the path to the .npz file where your predictions for the given sample are to `lidarseg_preds_bin_path` (Note the path name is correct as we share these arguments with `nuscenes-lidarseg` predictions) for these functions:\n",
    "- `list_lidarseg_categories`\n",
    "- `render_sample_data`\n",
    "- `render_pointcloud_in_image`\n",
    "- `render_sample`                 \n",
    "\n",
    "For example, let's assume the predictions for `my_sample` is stored at `/data/sets/nuscenes/panoptic/v1.0-mini` with the format `<lidar_sample_data_token>_panoptic.npz`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "my_sample = nusc.sample[87]\n",
    "sample_data_token = my_sample['data']['LIDAR_TOP']\n",
    "my_predictions_bin_file = os.path.join('/data/sets/nuscenes/panoptic/v1.0-mini', sample_data_token + '_panoptic.npz')\n",
    "\n",
    "nusc.render_pointcloud_in_image(my_sample['token'],\n",
    "                                pointsensor_channel='LIDAR_TOP',\n",
    "                                camera_channel='CAM_BACK',\n",
    "                                render_intensity=False,\n",
    "                                show_lidarseg=False,\n",
    "                                filter_lidarseg_labels=[17,22, 23, 24],\n",
    "                                show_lidarseg_legend=True,\n",
    "                                lidarseg_preds_bin_path=my_predictions_bin_file,\n",
    "                                show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these functions that render an entire scene, you will need to pass the path to the folder which contains the .npz files for each sample in a scene to `lidarseg_preds_folder`:\n",
    "- `render_scene_channel_lidarseg`\n",
    "- `render_scene_lidarseg`\n",
    "\n",
    "Pay special attention that **each set of predictions in the folder _must_ be a `.npz` file and named as `<lidar_sample_data_token>_panoptic.npz`**.\n",
    "\n",
    "(Note: the following code is commented out as it crashes in Jupyter notebooks.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# my_scene = nusc.scene[0]\n",
    "# my_folder_of_predictions = '/data/sets/nuscenes/panoptic/v1.0-mini'\n",
    "\n",
    "# nusc.render_scene_channel_lidarseg(my_scene['token'], \n",
    "#                                    'CAM_BACK', \n",
    "#                                    filter_lidarseg_labels=[9, 18, 24, 28],\n",
    "#                                    verbose=True, \n",
    "#                                    imsize=(1280, 720),\n",
    "#                                    lidarseg_preds_folder=my_folder_of_predictions,\n",
    "#                                    show_panoptic=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "And this brings us to the end of the tutorial for nuScenes-lidarseg and Panoptic nuScenes, enjoy!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}