LoFTR: Detector-Free Local Feature Matching with Transformers
Jiaming Sun*, Zehong Shen*, Yu'ang Wang*, Hujun Bao, Xiaowei Zhou
CVPR 2021
- Inference code and pretrained models (DS and OT) (2021-4-7)
- Code for reproducing the test-set results (2021-4-7)
- Webcam demo to reproduce the result shown in the GIF above (2021-4-13)
- Training code and training data preparation (expected 2021-6-10)
Discussions about the paper are welcomed in the discussion panel.
🤔 FAQ
- Undistorted images from D2Net are not available anymore.
For a temporal alternative, please use the undistorted images provided by the MegaDepth_v1 (should be downloaded along with the required depth files). We numerically compared these images and only found very subtle difference.
🚩 Updates
- Check out QuadTreeAttention, a new attention machanism that improves the efficiency and performance of LoFTR with less demanding GPU requirements for training.
- ✅ Integrated to Huggingface Spaces with Gradio. See Gradio Web Demo
Want to run LoFTR with custom image pairs without configuring your own GPU environment? Try the Colab demo:
LoFTR is integrated into kornia library since version 0.5.11.
pip install kornia
Then you can import it as
from kornia.feature import LoFTR
See tutorial on using LoFTR from kornia here.
# For full pytorch-lightning trainer features (recommended)
conda env create -f environment.yaml
conda activate loftr
# For the LoFTR matcher only
pip install torch einops yacs kornia
We provide the download link to
- the scannet-1500-testset (~1GB).
- the megadepth-1500-testset (~600MB).
- 4 pretrained models of indoor-ds, indoor-ot, outdoor-ds and outdoor-ot (each ~45MB).
By now, the environment is all set and the LoFTR-DS model is ready to go! If you want to run LoFTR-OT, some extra steps are needed:
[Requirements for LoFTR-OT]
We use the code from SuperGluePretrainedNetwork for optimal transport. However, we can't provide the code directly due its strict LICENSE requirements. We recommend downloading it with the following command instead.
cd src/loftr/utils
wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/superglue.py
[code snippets]
from src.loftr import LoFTR, default_cfg
# Initialize LoFTR
matcher = LoFTR(config=default_cfg)
matcher.load_state_dict(torch.load("weights/indoor_ds.ckpt")['state_dict'])
matcher = matcher.eval().cuda()
# Inference
with torch.no_grad():
matcher(batch) # batch = {'image0': img0, 'image1': img1}
mkpts0 = batch['mkpts0_f'].cpu().numpy()
mkpts1 = batch['mkpts1_f'].cpu().numpy()
An example is given in notebooks/demo_single_pair.ipynb
.
Run the online demo with a webcam or video to reproduce the result shown in the GIF above.
cd demo
./run_demo.sh