➡️ For state-of-the-art visual localization and SfM, checkout our new toolbox hloc:
cvg/Hierarchical-Localization 🔥

This repository accompanies our CVPR 2019 paper From Coarse to Fine: Robust Hierarchical Localization at Large Scale. We introduce a 6-DoF visual localization method that is accurate, scalable, and efficient, using HF-Net, a monolithic deep neural network for descriptor extraction. The proposed solution achieves state-of-the-art accuracy on several large-scale public benchmarks while running in real-time.

The proposed approach won the visual localization challenge of the CVPR 2019 workshop on Long-Term Visual Localization using this codebase. We also provide trained weights for HF-Net and reconstructed SfM 3D models.

Our method is significantly more robust, accurate, and scalable than standard approaches based on direct matching.

This code allows to:

• Perform state-of-the-art 6-DoF hierarchical localization using a flexible Python pipeline
• Train HF-Net with multi-task distillation in TensorFlow
• Evaluate feature detectors and descriptors on standard benchmarks
• Build Structure-from-Motion models based on state-of-the-art learned features

Python 3.6 is required. It is advised to run the following command within a virtual environment. By default, TensorFlow 1.12 GPU will be installed. You will be prompted to provide the path to a data folder (subsequently referred as $DATA_PATH) containing the datasets and pre-trained models and to an experiment folder ($EXPER_PATH) containing the trained models, training and evaluation logs, and CNN predictions. Create them wherever you wish and make sure to provide absolute paths. PyTorch 0.4.1 is also required to run the original SuperPoint and perform GPU-accelerated feature matching.

make install  # install Python requirements, setup paths

Refer to our dataset documentation for an overview of the supported datasets and their expected directory structure.

We provide a minimal example of the inference and localization with HF-Net in demo.ipynb. Download the trained model here and unpack it in $EXPER_PATH/saved_models/.

HF-Net simultaneously computes global descriptors and local features with an efficient architecture.

We provide code to perform and evaluate our hierarchical localization on the three challenging benchmark datasets of Sattler et al: Aachen Day-Night, RobotCar Seasons, and CMU Seasons.

Download the datasets as indicated in the dataset documentation. SfM models of Aachen, RobotCar, CMU, and Extended CMU, built SuperPoint and usable with HF-Net, are provided here. Download and unpack the HF-Net weights in $EXPER_PATH/hfnet/. To localize with NV+SP, download the network weights of NetVLAD and SuperPoint and put them in $DATA_PATH/weights/.

We first export the local features and global descriptors for all database and query images as .npz files. For the sake of flexibility, local descriptors are exported as dense maps for database images, but as sparse samples for query images.

For HF-Net or SuperPoint:

python3 hfnet/export_predictions.py \
	hfnet/configs/[hfnet|superpoint]_export_[aachen|cmu|robotcar]_db.yaml \
	[superpoint/][aachen|cmu|robotcar] \
	[--exper_name hfnet] \ # for HF-Net only
	--keys keypoints,scores,local_descriptor_map[,global_descriptor]
python3 hfnet/export_predictions.py \
	hfnet/configs/[hfnet|superpoint]_export_[aachen|cmu|robotcar]_queries.yaml \
	[superpoint/][aachen|cmu|robotcar] \
	[--exper_name hfnet] \ # for HF-Net only
	--keys keypoints,scores,local_descriptors[,global_descriptor]

For HF-net in HPatches:

python hfnet/export_predictions_hfnet.py hfnet/configs/hfnet_export_hpatches.yaml hfnetV1_hpatches_predictions --keys global_descriptor,keypoints,local_descriptors --as_dataset

For SuperPoint in HPatches

python3 hfnet/export_predictions.py hfnet/configs/superpoint_export_hpatches.yaml superpoint_hpatches_predictions --keys keypoints,scores,local_descriptors --as_dataset

For NetVLAD:

python3 hfnet/export_predictions.py \
	hfnet/configs/netvlad_export_[aachen|cmu|robotcar].yaml \
	netvlad/[aachen|cmu|robotcar] \
	--keys global_descriptor

For Aachen:

python3 hfnet/evaluate_aachen.py \
	<sfm_model_name_or_path> \
	<eval_name>_[night|day] \
	--local_method [hfnet|superpoint|sift] \
	--global_method [hfnet|netvlad] \
	--build_db \
	--queries [night_time|day_time] \
	--export_poses

For RobotCar:

python3 hfnet/evaluate_robotcar.py \
	<sfm_model_name_or_path> \
	<eval_name> \
	--local_method [hfnet|superpoint|sift] \
	--global_method [hfnet|netvlad] \
	--build_db \
	--queries [dusk|sun|night|night-rain] \
	--export_poses

For CMU:

python3 hfnet/evaluate_cmu.py \
	<sfm_model_name_or_path> \
	<eval_name> \
	--local_method [hfnet|superpoint|sift] \
	--global_method [hfnet|netvlad] \
	--build_db \
	--slice [2|3|4|5|6|7|8|9|10|17] \
	--export_poses

The localization parameters can be adjusted in hfnet/evaluate_[aachen|robotcar|cmu].py. The evaluation logs and estimated poses are written to $EXPER_PATH/eval/[aachen|robotcar|cmu]/<eval_name>*. Of particular interest are the PnP+RANSAC success rate, the average number of inliers per query, and the average inlier ratio.

Successful and failed queries can be visualized in notebooks/visualize_localization_[aachen|robotcar|cmu].ipynb.

Instructions to train HF-Net are provided in the training documentation.

Instructions to evaluate feature detectors and descriptors on the HPatches and SfM datasets are provided in the local evaluation documentation.

Instructions and scripts to build SfM models using COLMAP for any learned features are provided in colmap-helpers.

Please consider citing the corresponding publication if you use this work in an academic context:

@inproceedings{sarlin2019coarse,
  title={From Coarse to Fine: Robust Hierarchical Localization at Large Scale},
  author={Sarlin, Paul-Edouard and Cadena, Cesar and Siegwart, Roland and Dymczyk, Marcin},
  booktitle={CVPR},
  year={2019}
}