Added the source code

DATASETS.md

@@ -0,0 +1,77 @@
+## Datasets
+We recommend to setup the datasets in `$LANEATT_ROOT/datasets` (it can be a symlink), where `$LANEATT_ROOT` is the code's root directory. All the configs provided in this repository expect the datasets to be in this path and changing it will require you to update the configs accordingly.
+#### TuSimple
+[\[Website\]](https://github.com/TuSimple/tusimple-benchmark/tree/master/doc/lane_detection)
+[\[Download page\]](https://github.com/TuSimple/tusimple-benchmark/issues/3)
+
+**How to set up**
+
+Inside the code's root directory, run the following:
+```bash
+mkdir datasets # if it does not already exists
+cd datasets
+# train & validation data (~10 GB)
+mkdir tusimple
+wget "https://s3.us-east-2.amazonaws.com/benchmark-frontend/datasets/1/train_set.zip"
+unzip train_set.zip -d tusimple
+# test images (~10 GB)
+mkdir tusimple-test
+wget "https://s3.us-east-2.amazonaws.com/benchmark-frontend/datasets/1/test_set.zip"
+unzip test_set.zip -d tusimple-test
+# test annotations
+wget "https://s3.us-east-2.amazonaws.com/benchmark-frontend/truth/1/test_label.json" -P tusimple-test/
+cd ..
+```
+#### CULane
+
+[\[Website\]](https://xingangpan.github.io/projects/CULane.html)
+[\[Download page\]](https://drive.google.com/open?id=1mSLgwVTiaUMAb4AVOWwlCD5JcWdrwpvu)
+
+**How to set up**
+
+Inside the code's root directory, run the following:
+```bash
+mkdir datasets # if it does not already exists
+cd datasets
+mkdir culane
+# train & validation images (~30 GB)
+gdown "https://drive.google.com/uc?id=1AQjQZwOAkeBTSG_1I9fYn8KBcxBBbYyk"
+gdown "https://drive.google.com/uc?id=1PH7UdmtZOK3Qi3SBqtYOkWSH2dpbfmkL"
+gdown "https://drive.google.com/uc?id=14Gi1AXbgkqvSysuoLyq1CsjFSypvoLVL"
+tar xf driver_23_30frame.tar.gz
+tar xf driver_161_90frame.tar.gz
+tar xf driver_182_30frame.tar.gz
+# test images (~10 GB)
+gdown "https://drive.google.com/uc?id=1LTdUXzUWcnHuEEAiMoG42oAGuJggPQs8"
+gdown "https://drive.google.com/uc?id=1daWl7XVzH06GwcZtF4WD8Xpvci5SZiUV"
+gdown "https://drive.google.com/uc?id=1Z6a463FQ3pfP54HMwF3QS5h9p2Ch3An7"
+tar xf driver_37_30frame.tar.gz
+tar xf driver_100_30frame.tar.gz
+tar xf driver_193_90frame.tar.gzt
+# all annotations (train, val and test)
+gdown "https://drive.google.com/uc?id=1QbB1TOk9Fy6Sk0CoOsR3V8R56_eG6Xnu"
+tar xf annotations_new.tar.gz
+gdown "https://drive.google.com/uc?id=18alVEPAMBA9Hpr3RDAAchqSj5IxZNRKd"
+tar xf list.tar.gz
+```
+#### LLAMAS
+[\[Website\]](https://unsupervised-llamas.com/llamas/)
+[\[Download page\]](https://unsupervised-llamas.com/llamas/login/?next=/llamas/download)
+
+An account in the website is required to download the dataset.
+
+**How to set up**
+1. Download the set of color images (`color_images.zip`, 108 GB)
+2. Download the annotations (`labels.zip`, 650 MB)
+3. Unzip both files (`color_images.zip` and `labels.zip`) into the same directory (e.g., `datasets/llamas/`), which will be the dataset's root. This should result in a directory structure like that:
+```
+llamas
+├── color_images
+│   ├── test
+│   ├── train
+│   └── valid
+└── labels
+ ├── train
+ └── valid
+```
+

README.md

@@ -1,12 +1,164 @@
+<div align="center">
+
 # LaneATT
 [![arXiv](https://img.shields.io/badge/arXiv-2010.12035-b31b1b.svg)](https://arxiv.org/abs/2010.12035)
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/keep-your-eyes-on-the-lane-attention-guided/lane-detection-on-culane)](https://paperswithcode.com/sota/lane-detection-on-culane?p=keep-your-eyes-on-the-lane-attention-guided)
+![Method overview](data/figures/method-overview.png "Method overview")
+</div>
+
+This repository holds the source code for LaneATT, a novel state-of-the-art lane detection model proposed in the [paper](https://arxiv.org/abs/2010.12035) "_Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection_", by [Lucas Tabelini](https://github.com/lucastabelini), [Rodrigo Berriel](http:https://rodrigoberriel.com), [Thiago M. Paixão](https://sites.google.com/view/thiagopx), [Claudine Badue](http:https://www.inf.ufes.br/~claudine/), [Alberto F. De Souza](http:https://www.lcad.inf.ufes.br/team/index.php/Prof._Dr._Alberto_Ferreira_De_Souza), and [Thiago Oliveira-Santos](http:https://www.inf.ufes.br/~todsantos/home).
+
+### Table of contents
+1. [Prerequisites](#1-prerequisites)
+2. [Install](#2-install)
+3. [Getting started](#3-getting-started)
+4. [Results](#4-results)
+5. [Code structure](#5-code-structure)
+6. [Citation](#6-Citation)
+
+
+### 1. Prerequisites
+- Python >= 3.5
+- PyTorch == 1.6, tested on CUDA 10.2. The models were trained and evaluated on PyTorch 1.6. When testing with other versions, the results (metrics) are slightly different.
+- CUDA, to compile the NMS code
+- Other dependencies described in `requirements.txt`
+
+The versions described here were the lowest the code was tested with. Therefore, it may also work in other earlier versions, but it is not guaranteed (e.g., the code might run, but with different outputs).
+
+### 2. Install
+Conda is not necessary for the installation, as you can see, I only use it for PyTorch and Torchvision.
+Nevertheless, the installation process here is described using it.
+
+```bash
+conda create -n laneatt python=3.8 -y
+conda activate laneatt
+conda install pytorch==1.6 torchvision -c pytorch
+pip install -r requirements.txt
+cd lib/nms; python setup.py install; cd -
+```
+
+### 3. Getting started
+#### Datasets
+For a guide on how to download and setup each dataset, see [DATASETS.md](DATASETS.md).
+
+#### Training & testing
+Train a model:
+```
+python main.py train --exp_name example --cfg example.yml
+```
+For example, to train LaneATT with the ResNet-34 backbone on TuSimple, run:
+```
+python main.py train --exp_name laneatt_r34_tusimple --cfg cfgs/laneatt_tusimple_resnet34.yml
+```
+After running this command, a directory `experiments` should be created (if it does not already exists). Another
+directory `laneatt_r34_tusimple` will be inside it, containing data related to that experiment (e.g., model checkpoints, logs, evaluation results, etc)
+
+Evaluate a model:
+```
+python main.py test --exp_name example
+```
+This command will evaluate the model saved in the last checkpoint of the experiment `example` (inside `experiments`).
+If you want to evaluate another checkpoint, the `--epoch` flag can be used. For other flags, please see `python main.py -h`.
+
+#### Reproducing a result from the paper
+0. Set up the dataset you want to reproduce the results on (as described in [DATASETS.md](DATASETS.md)).
+1. Download the zip containing all pretrained models and then unzip it at the code's root:
+```bash
+gdown "https://drive.google.com/uc?id=1R638ou1AMncTCRvrkQY6I-11CPwZy23T" # main experiments on TuSimple, CULane and LLAMAS (1.3 GB)
+unzip laneatt_experiments.zip
+```
+2. Run the evaluation (inference + metric computation):
+```bash
+python main.py test --exp_name $EXP_NAME
+```
+Replace `$EXP_NAME` with the name of a directory inside `experiments/`. For instance, if you want to reproduce the results using the ResNet-34 backbone on the TuSimple dataset, run:
+```bash
+python main.py test --exp_name laneatt_r34_tusimple
+```
+The results on TuSimple and LLAMAS should match exactly the ones reported in the paper. The results on CULane will deviate in the order of 0.1% (as shown in the CULane table below), since the metric reported on the paper was computed with the official code (C++), while this script will compute it using our implementation (which is much faster and in Python). The official metric implementation is available [here](https://github.com/XingangPan/SCNN/tree/master/tools/lane_evaluation).
+
+### 4. Results
+![F1 vs. Latency for state-of-the-art methods on lane detection](data/figures/f1-vs-latency.png "F1 vs. Latency for state-of-the-art methods on lane detection")
+
+#### CULane
+
+| Backbone | F1, official impl. (%) | F1, our impl. (%) | FPS |
+| :--- | ---: | ---: | ---:|
+| ResNet-18 | 75.13 | 75.08 | 250 |
+| ResNet-34 | 76.68 | 76.66 | 171 |
+| ResNet-122 | 77.02 | 77.02 | 26 |
+
+"F1, official impl." refers to the official CULane metric implementation in C++. "F1, our impl" refers to our implementation of the metric in Python. The results reported in the paper were computed using the [official metric implementation](https://github.com/XingangPan/SCNN/tree/master/tools/lane_evaluation)
+ (requires OpenCV 2.4).
+ [![CULane video](data/figures/culane_video.png "CULane video")](https://youtu.be/ghs93acwkBQ)
+
+#### TuSimple
+| Backbone | Accuracy (%) | FDR (%) | FNR (%) | F1 (%) | FPS |
+| :--- | ---: | ---: | ---: | ---: | ---:|
+| ResNet-18 | 95.57 | 3.56 | 3.01 | 96.71 | 250 |
+| ResNet-34 | 95.63 | 3.53 | 2.92 | 96.77 | 171 |
+| ResNet-122 | 96.10 | 4.64 | 2.17 | 96.06 | 26 |
+
+Since the TuSimple dataset is not sequential, no qualitative video is available.
+
+#### LLAMAS
+| Backbone | F1 (%) | Precision (%) | Recall (%) | FPS |
+| :--- | ---: | ---: | ---: | ---:|
+| ResNet-18 | 93.46 | 96.92 | 90.24 | 250 |
+| ResNet-34 | 93.74 | 96.79 | 90.88 | 171 |
+| ResNet-122 | 93.54 | 96.82 | 90.47 | 26 |
 
-This repository will hold the source code for LaneATT, a novel state-of-the-art lane detection model proposed in the [paper](https://arxiv.org/abs/2010.12035) "_Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection_", by [Lucas Tabelini](https://github.com/lucastabelini), [Rodrigo F. Berriel](http:https://rodrigoberriel.com), [Thiago M. Paixão](https://sites.google.com/view/thiagopx), [Claudine Badue](https://www.inf.ufes.br/~claudine/), [Alberto F. De Souza](https://inf.ufes.br/~alberto), and [Thiago Oliveira-Santos](https://www.inf.ufes.br/~todsantos/home).
+ [![LLAMAS video](data/figures/llamas_video.png "LLAMAS video")](https://youtu.be/1f_y4A-muMg)
 
-**The code will be published soon. Subscribe to [this issue](https://github.com/lucastabelini/LaneATT/issues/1) to be notified when that happens.**
+Additional results can be seen in the paper.
 
-![LaneATT overview](figures/laneatt_overview.png "LaneATT overview")
+### 5. Code structure
+- **cfgs:** Default configuration files
+- **figures:** Images used in this repository
+- **lib**
+ - **datasets**
+ - **culane.py:** CULane annotation loader
+ - **lane_dataset.py:** Transforms raw annotations from a `LaneDatasetLoader` into a format usable by the model
+ - **lane_dataset_loader.py:** Abstract class that each dataset loader implements
+ - **llamas.py:** LLAMAS annotation loader
+ - **nolabel_dataset.py:** Used on data with no annotation available (or quick qualitative testing)
+ - **tusimple.py:** TuSimple annotation loader
+ - **models:**
+ - **laneatt.py:** LaneATT implementation
+ - **matching.py:** Utility function for ground-truth and proposals matching
+ - **resnet.py:** Implementation of ResNet
+ - **nms:** NMS implementation
+ - **config.py:** Configuration loader
+ - **experiment.py:** Tracks and stores information about each experiment
+ - **focal_loss.py:** Implementation of Focal Loss
+ - **lane.py:** Lane representation
+ - **runner.py:** Training and testing loops
+- **utils**:
+ - **culane_metric.py:** Unofficial implementation of the CULane metric. This implementation is faster than the oficial,
+ however, it does not matches exactly the results of the official one (error in the order of 1e-4). Thus, it was used only during the model's development.
+ For the results reported in the paper, the official one was used.
+ - **gen_anchor_mask.py**: Computes the frequency of each anchor in a dataset to be used in the anchor filtering step
+ - **gen_video.py:** Generates a video from a model's predictions
+ - **llamas_metric.py**: Official implementation of the LLAMAS metric
+ - **llamas_utils.py**: Utilities functions for the LLAMAS dataset
+ - **speed.py:** Measure efficiency-related metrics of a model
+ - **tusimple_metric.py**: Official implementation of the TuSimple metric
+ - **viz_dataset.py**: Show images sampled from a dataset (post-augmentation)
+- **main.py:** Runs the training or testing phase of an experiment
 
-### Abstract
+### 6. Citation
+If you use this code in your research, please cite:
 
-Modern lane detection methods have achieved remarkable performances in complex real-world scenarios, but many have issues maintaining real-time efficiency, which is important for autonomous vehicles. In this work, we propose LaneATT: an anchor-based deep lane detection model, which, akin to other generic deep object detectors, uses the anchors for the feature pooling step. Since lanes follow a regular pattern and are highly correlated, we hypothesize that in some cases global information may be crucial to infer their positions, especially in conditions such as occlusion, missing lane markers, and others. Thus, this work proposes a novel anchor-based attention mechanism that aggregates global information. The model was evaluated extensively on three of the most widely used datasets in the literature. The results show that our method outperforms the current state-of-the-art methods showing both higher efficacy and efficiency. Moreover, an ablation study is performed along with a discussion on efficiency trade-off options that are useful in practice.
+```bibtex
+@misc{laneatt2020arxiv,
+ author = {Lucas Tabelini
+ and Rodrigo Berriel
+ and Thiago M. Paix\~ao
+ and Claudine Badue
+ and Alberto Ferreira De Souza
+ and Thiago Oliveira-Santos},
+ title = {{Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection}},
+ eprint = {arXiv:2010.12035},
+ year = {2020}
+}
+```

cfgs/laneatt_culane_resnet122.yml

@@ -0,0 +1,84 @@
+# Model settings
+val_every: 1000
+model_checkpoint_interval: 1
+seed: 0
+model:
+ name: LaneATT
+ parameters:
+ backbone: resnet122
+ S: &S 72
+ topk_anchors: 1000
+ anchors_freq_path: 'data/culane_anchors_freq.pt'
+ img_h: &img_h 360
+ img_w: &img_w 640
+batch_size: 4
+epochs: 15
+loss_parameters: {}
+train_parameters:
+ conf_threshold:
+ nms_thres: 15.
+ nms_topk: 3000
+test_parameters:
+ conf_threshold: 0.5
+ nms_thres: 50.
+ nms_topk: &max_lanes 4
+optimizer:
+ name: Adam
+ parameters:
+ lr: 0.0003
+lr_scheduler:
+ name: CosineAnnealingLR
+ parameters:
+ T_max: 333300 # 15 * 22220 iterations
+
+# Dataset settings
+datasets:
+ train:
+ type: LaneDataset
+ parameters:
+ S: *S
+ dataset: culane
+ split: train
+ img_size: [*img_h, *img_w]
+ max_lanes: *max_lanes
+ normalize: false
+ aug_chance: 1.0
+ augmentations:
+ - name: Affine
+ parameters:
+ translate_px:
+ x: !!python/tuple [-25, 25]
+ y: !!python/tuple [-10, 10]
+ rotate: !!python/tuple [-6, 6]
+ scale: !!python/tuple [0.85, 1.15]
+ - name: HorizontalFlip
+ parameters:
+ p: 0.5
+
+ root: "datasets/culane"
+
+ test:
+ type: LaneDataset
+ parameters:
+ S: *S
+ dataset: culane
+ split: test
+ img_size: [*img_h, *img_w]
+ max_lanes: *max_lanes
+ normalize: false
+ aug_chance: 0
+ augmentations:
+ root: "datasets/culane"
+
+ val:
+ type: LaneDataset
+ parameters:
+ S: *S
+ dataset: culane
+ split: val
+ img_size: [*img_h, *img_w]
+ max_lanes: *max_lanes
+ normalize: false
+ aug_chance: 0
+ augmentations:
+ root: "datasets/culane"