This code implements the event-based visual odometry pipeline described in the paper EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real-time by Henri Rebecq, Timo Horstschaefer, Guillermo Gallego and Davide Scaramuzza.
If you use any of this code, please cite the following publications:
@Article{RebecqEVO,
author = {Rebecq, Henri and Horstschaefer, Timo and Gallego, Guillermo and Scaramuzza, Davide},
journal = {IEEE Robotics and Automation Letters},
title = {EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time},
year = {2017},
volume = {2},
number = {2},
pages = {593-600},
doi = {10.1109/LRA.2016.2645143}
}
@InProceedings{Gehrig_2020_CVPR,
author = {Daniel Gehrig and Mathias Gehrig and Javier Hidalgo-Carri\'o and Davide Scaramuzza},
title = {Video to Events: Recycling Video Datasets for Event Cameras},
booktitle = {{IEEE} Conf. Comput. Vis. Pattern Recog. (CVPR)},
month = {June},
year = {2020}
}
The code comes with a test dataset. If you don't have an event camera and want to run the code on more data, you can do one of the following:
- Using one of our event camera datasets:
- Generate one using event-camera simulator: https://rpg.ifi.uzh.ch/esim/index.html
- Convert a standard video to events using our "Video to Events: Recycling Video Datasets for Event Cameras." cited above, code here: https://github.com/uzh-rpg/rpg_vid2e
-
The proposed EVO method is patented.
H. Rebecq, G. Gallego, D. Scaramuzza Simultaneous Localization and Mapping with an Event Camera Pub. No.: WO/2018/037079. International Application No.: PCT/EP2017/071331
-
The license is available here.
The open sourcing was curated by Antonio Terpin and Daniel Gehrig.
Remark that this is research code, any fitness for a particular purpose is disclaimed.
This software depends on ROS. Installation instructions can be found here. We have tested this software on Ubuntu 18.04 and ROS Melodic.
-
Create and initialize a new catkin workspace if needed
mkdir -p ~/catkin_ws/src && cd ~/catkin_ws/ catkin config \ --init --mkdirs --extend /opt/ros/melodic \ --merge-devel --cmake-args \ -DCMAKE_BUILD_TYPE=Release
-
Clone this repository
cd src/ && git clone [email protected]:uzh-rpg/rpg_dvs_evo_open.git
-
Clone (and fix) dependencies
./rpg_dvs_evo_open/install.sh [ros-version] # [ros-version]: melodic, ..
Substitute
[ros-version]
with your actual ROS distribution. For instance:./rpg_dvs_evo_open/install.sh melodic
Make sure to install ROS and the commonly used packages (such as rviz, rqt, ..).
The above commands do the following:
-
First, we install the required packages.
-
Second, we clone the repositories evo relies on.
Remark that with the above commands we install also the dependencies required to run the pipeline live. If you do not need them, you can comment the unnecessary packages from the dependencies.yaml file (davis driver).
Please refer to this repo if there are issues with the driver installation (required for the live).
-
-
Build the packages
catkin build dvs_tracking
Building everything might take some time... this might be the right time for a coffee! :)
... and do not forget to source the workspace afterwards!
source ../devel/setup.bash
Example | Launch file | Rosbag |
---|---|---|
Multi-keyframe sequence | flyingroom.launch | 538 MB |
Desk sequence | desk.launch | 145 MB |
To download the rosbags, you can use the following command:
wget [rosbag link] -O /path/to/download/folder/
For instance, the following will download the multi-keyframe sequence rosbag:
wget https://rpg.ifi.uzh.ch/data/EVO/code_examples/evo_flyingroom.bag -O /tmp/evo_flyingroom.bag
To run the pipeline from a rosbag, first start the pipeline as
roslaunch dvs_tracking [launch-file] auto_trigger:=true
where the specific launch file (fine-tuned) for each example is listed in the following table. The most interesting and repeatible one (due to the bootstrapping sequence, see further improvements) is the multi-keyframe sequence.
Then, once the everything is loaded, run the rosbag as
rosbag play -r 0.7 [rosbag-file]
For instance,
rosbag play -r 0.7 /tmp/evo_flyingroom.bag
Remark that we set the auto_trigger
parameter to true. You can also set it to false and follow the instruction on how to run it live.
If anything fails, just try it again (give it a couple of chances!), and make sure to follow exactly the above instructions.
In the further improvements section are outlined the things that might go wrong when running the code. To improve the reliability when playing the rosbags (running live is easier), consider using -r .7
, to reduce the rate. This helps especially when the hardware is not powerful enough.
For instance,
rosbag play /tmp/evo_flyingroom -r .7
Eventually, setting bootstrap_image_topic:=/dvs/image_raw
will bootstrap from traditional frames and later switch to only events. This is the most reliable way currently available to bootstrap.
roslaunch dvs_tracking [launch-file] bootstrap_image_topic:=/dvs/image_raw auto_trigger:=true
The procedure is analogous to the one explained in the examples:
-
Run the ros core on the first terminal.
roscore
-
On a second terminal, launch the event camera driver.
rosrun davis_ros_driver davis_ros_driver
-
On another terminal, launch the pipeline, disabling the auto-triggering.
roslaunch dvs_tracking live.launch auto_trigger:=false camera_name:=[calibration filename] events_topic:=[events topic]
For instance:
roslaunch dvs_tracking live.launch auto_trigger:=false camera_name:=DAVIS-ijrr events_topic:=/dvs/events
If your calibrations filenames are
my_camera.yaml
, then usecamera_name:=my_camera
. Make sure to use the same name for both calibration files. If your sensor outputs events under the topic/my_sensor/events
, then useevents_topic:=/my_sensor/events
.For the SVO based bootstrapper, proceed with step 4. For the fronto-planar bootstrapper, go to step 5. How to set up the fronto-planar bootstrapper is explained in the live.launch file.
If you want to bootstrap from traditional frames, you can use the command:
roslaunch dvs_tracking live.launch bootstrap_image_topic:=[topic raw frames] auto_trigger:=false
For instance
bootstrap_image_topic:=/dvs/image_raw
.In this case, it makes sense to use the SVO-based bootstrapping only. This option is recommended to debug/improve/extend the rest of the EVO pipeline, without worrying about the quality of the bootstrapping.
-
You should see two rqt GUI. One is the SVO GUI. Reset and start the pipeline until it tracks decently well. Make sure to set the namespace to
svo
. -
Press the
Bootstrap
button in the EVO GUI. This will automatically trigger the pipeline.Alternatively, it is also possible to trigger one module at the time:
- Press the
Start/Reset
button inrqt_evo
. Perform a circle (or more), and then pressUpdate
. This will trigger a map creation. - If the map looks correct, press
Switch to tracking
to start tracking with EVO. If not, reiterate the map creation. - As the camera moves out of the current map, the latter will be automatically updated if the
Map-Expansion
is enabled. You may disableMap-Expansion
to track high-speed motions using the current map (single keyframe tracking). - The scene should have enough texture and the motions should recall the ones that you can see in the provided examples.
- Press the
-
If anything fails, just press
Ctrl+C
and restart the live node ;)
Some remarks:
- The calibration files paths will be built as
$(find dvs_tracking)/parameters/calib/ncamera/$(arg camera_name).yaml
and$(find dvs_tracking)/parameters/calib/$(arg camera_name).yaml
, wherecamera_name
is specified as argument to the launch file. You can also set it as default inlive.launch
. - If your sensor provides frames under a topic
/my_sensor/image_raw
, and you want to bootstrap from the traditional frames, you can usebootstrap_image_topic:=/my_sensor/image_raw
.
To run the pipeline live, first adjust the template live.launch to your sensor and scene. You can follow the following steps. Further customization, such as which bootstrapper to use, are explained in the launch file itself.
Make sure you have updated calibration files for your event camera, in the dvs_tracking/parameters/calib
folder.
Make sure your .yaml
files have the same format as the provided ones (single camera format, multiple cameras format).
Remark that we have two calibration files in two different format for the same sensor. The single camera format has to be placed in the dvs_tracking/parameters/calib
folder and the multiple cameras format in the dvs_tracking/parameters/calib/ncamera
folder.
See this section for further references on calibration.
Adjust the parameters in the launch file dvs_tracking/launch/template.launch
.
Tuning is crucial for a good performance of the pipeline, in particular the min_depth
and max_depth
parameters in the mapping node, and the bootstrap
node parameters.
An explanation of all the parameters for each module of the pipeline can be found in the Wiki.
The main parameters can be found in the template launch file, and are explained contextually. We still invite you to have a look at the Wiki, to discover further interesting features ;)
Module |
---|
Global parameters |
Bootstrapping |
Mapping |
Tracking |
If you are not using the fronto-planar bootstrapper, then you might need to tune SVO.
Remark that this might not be needed. You can test the svo tuning bootstrapping from traditional frames:
roslaunch dvs_tracking live.launch bootstrap_image_topic:=[topic of raw frames] auto_trigger:=[true/false]
For instance, bootstrap_image_topic:=/dvs/image_raw
.
In the following we outline the main problems currently known and possible remedies. You are very welcome to contribute to this pipeline to make it even better!
What can go wrong | TODO |
---|---|
Randomness due to OS scheduler, unreliable rosbags | Implement rosbag data provider pattern, and ensure correctness of events consumption |
Currently the pipeline uses multiple nodes. Switching to nodelets or using a single node could improve the repeatability of the rosbags. | |
Robustness | bootstrapping: catch whenever SVO does not converge and trigger an automatic restart (what a human operator would eventually do manually). |
tracking: catch whenever the tracker diverges, and re-initialize. Currently we have two parameters to predict this situation, namely min_map_size and min_n_keypoints . |
|
Improve bootstrapping robustness | Currently we have two working ways to bootstrap the pipeline from events: from SVO (feeding it with events frames) and with a fronto-planar assumption. |
Reducing the assumptions required and making them more reliable would allow a better bootstrapping, reducing the gap to the bootstrapping from traditional frames (bootstrap_image_topic:=/dvs/image_raw ). |