This is a freely available hardware and software prototype demonstrating our research activities concerning processing of video signals from human skin. We apply this technique in the context of ultra-high-field MRI for heart rate measurement and image acquisition but other scenarios are possible (e.g. gaming, sports).
- Feb. 2017: I am pleased to announce that this project will be presented at the ISMRM 25th Annual Meeting & Exhibition as a Power Pitch presentation. Additionally, I present our recent investigation of local skin color phase variations.
- Dec 2016: Our initial study on video-based MRI triggering has been accepted recently for open-access publication in Biomedical Engineering Online.
The aim of our research is to overcome the limitations of contact-based hardware for MRI patient monitoring (e.g. electrocardiography, pulse oximetry) as they are error-prone, especially during long or ultra-high-field MRI examinations. Instead, we develop video-based (and therefore contact-free) real-time methods based on recent findings in remote vital sign measurement. A valuable overview of this field of research (unrelated to MRI) can be found in [1]. So far, journal papers within the context of MRI were published by a Stanford group [2] and our group [3].
The software presented here will be used to demonstrate some of the methods we have developed so far. Regarding video-based heart rate frequency estimation, there are other valuable open-source projects (e.g. webcam-pulse-detector) or commercial products (e.g. Philips Vital Signs Camera). This aspect is part of our past work [5]; however we are more interested in developing methods for estimating the current phase of the cardiac cycle accurately. See the literature list if you are interested in the scientific background.
Next to the software, you can find information on a low-cost device, based on an Arduino Nano, for sending triggers from the software to the MR scanner. This is needed, when applying algorithms for video-based cardiac gating. Additionally, you can find the schematics of an LED array for increasing illumination of the skin.
Proposed software and hardware can NOT be used for diagnosis or treatment. Results are estimated and for entertainment purposes only!
GNU GPL v3.0
Develop branch 
Master branch 
Latest release 
Use the release version if you want to use a manually tested and stable version. The master branch contains the most current version that should be stable and the develop branch contains the current bleeding-edge development version.
-
Read video stream from OpenCV compatible camera or read video stream from hard disk
-
Crop video to manually-defined ROI or use Viola-Jones algorithm for face detection [4]
-
Store frames from camera on hard disk
-
A virtual serial device is used if the trigger device is not available (Please note that the emulation decreases performance)
-
Heart rate estimation as described in [5]
-
Signal filtering as described in [6]
-
MRI triggering by phase information as described in [7]
| HR estimation | MRI triggering |
|---|---|
 |
 |
| Signal filtering | HR estimation |
 |
 |
Video Demonstration: You can find a video of v0.1-beta here: here
Important notice: For most accurate results, place your finger tip directly on the camera sensor (see screenshot 1). The higher the distance to the camera sensor, the lower the signal-to-noise ratio. If you want to obtain accurate results from remote skin, good illumination conditions and minimal subject motion is crucial. Additionally, there may be artefacts by other biosignals such as respiration. Comparing the results of heart rate estimation to a pulse oximeter from clinical practice (see screenshot 3), underlines the accuracy of the algorithm under adequate conditions. Additionally, we used the videos from the Eulerian Video Magnification website for evaluation (see screenshot 4).
- Python 2.7
- modules: scipy, numpy, matplotlib, pillow, pyserial, and several other modules
- see requirements.txt for detailed information
- TK toolkit python bindings (tkInter >= 2.7.5)
- OpenCV 2.4 python bindings
Run make (which is at the moment basically pip install -r requirements.txt and installs required packages via pip).
OpenCV/TK bindings have to be installed manually (e.g. by sudo apt-get install python-opencv python-tk).
cd src; python main.py
The current development branch is tested on Ubuntu 14.04 (and on Ubuntu 12.04 using Travis-CI). The current master branch is additionally tested on Windows 7 but the performance on Windows is inferior.
| Information | FPS | Duration | Resolution | Download |
|---|---|---|---|---|
| The finger of a volunteer was placed directly on the camera of an off-the-shelf smartphone (see screenshot 1). | 25 | 2:00 | 640x360 | here |
| The forehead of a volunteer undergoing MRI examination was recorded with a MRI-compatible camera (see screenshot 2). | 25 | 2:00 | 720x480 | here |
| A volunteer was recorded during office work using the webcam of an off-the-shelf business laptop. A pulse oximeter was applied as reference (see screenshot 3). | 30 | 1:00 | 640x480 | here |
More videos from persons in different situations can be found on the Eulerian Video Magnification website. More videos from subjects inside the MR bore can be found in the supplemental material of the paper by Maclaren et al. [2]
heart.png and heartbeat.png: Icons made by Madebyoliver from www.flaticon.com are licensed by CC 3.0 BY
Nicolai Spicher (https://fh-dortmund.de/spicher)
See website for email address and please use my PGP key.
Department of Computer Science, University of Applied Sciences and Arts Dortmund
[1] Sun Y. and Thakor N. Photoplethysmography Revisited: From Contact to Noncontact, From Point to Imaging IEEE Transactions on Biomedical Engineering, Vol. 63 (3), 2016 (PDF)
[2] Maclaren J., Aksoy M. and Bammer R. Contact-free physiological monitoring using a markerless optical system. Magnetic resonance in medicine. 74(2):571-7 2015. (PDF)
[3] Spicher N., Kukuk M., Maderwald S., Ladd ME. Initial evaluation of prospective cardiac triggering using photoplethysmography signals recorded with a video camera compared to pulse oximetry and electrocardiography at 7T MRI Biomedical Engineering Online. 15(1):126, 2016. (PDF)
[4] Viola P., Jones M. Rapid object detection using a boosted cascade of simple features Proceedings of the 2001 IEEE Computer Society Conference on on Computer Vision and Pattern Recognition, Kauai, USA, 08.-14.12.2001. (PDF)
[5] Spicher N., Maderwald S., Ladd ME. and Kukuk M. Heart rate monitoring in ultra-high-field MRI using frequency information obtained from video signals of the human skin compared to electrocardiography and pulse oximetry Proceedings of the 49th Annual Conference of the German Society for Biomedical Engineering, Luebeck, Germany, 16.-18.09.2015. (PDF)
[6] Spicher N., Maderwald S., Ladd ME. and Kukuk M. High-speed, contact- free measurement of the photoplethysmography waveform for MRI triggering Proceedings of the 24th Annual Meeting of the ISMRM, Singapore, Singapore, 07.05.-13.05.2016. (PDF)
[7] Spicher N., Kukuk M., Ladd ME. and Maderwald S. In vivo 7T MR imaging triggered by phase information obtained from video signals of the human skin Proceedings of the 23nd Annual Meeting of the ISMRM, Toronto, Canada, 30.05.-05.06.2015. (PDF)