Full movie: YouTube

This repository contains code for LEAP (LEAP Estimates Animal Pose), a framework for animal body part position estimation via deep learning.

Preprint: Pereira et al., bioRxiv (2018)

We are still working on documentation and preparing parts of the code. See the Features section below for an overview and status of each component.

• Tracking and alignment code
• Cluster sampling GUI
• Skeleton creation GUI (create_skeleton)
• GUI for labeling new dataset (label_joints)
• Network training through the labeling GUI
• MATLAB (predict_box.m) and Python (leap.training) interfaces for predicting on new data
• GUI for predicting on new data
• Training data + labels for main fly dataset used in analyses
• Trained network for predicting on main fly dataset
• Analysis/figure generation code
• Documentation
• Examples of usage

GUIs and analyses are implemented in MATLAB, but is not required for using the neural network functionality implemented in Python.

We use MATLAB R2018a with the following toolboxes: Parallel Computing Toolbox, Statistics and Machine Learning Toolbox, Computer Vision Toolbox, Image Processing Toolbox, Signal Processing Toolbox.

All MATLAB external toolboxes are included in the leap/toolbox subfolder. Just add the leap subdirectory to the MATLAB Search Path to access all functionality:

addpath(genpath('leap'))

All neural network and GPU functionality is implemented in Python. The library was designed to be easy to use by providing commandline interfaces, but it can also be used programatically if the MATLAB GUIs are not required.

For the Python environment, we recommend Anaconda 5.1.0 with Python 3.6.4.

The versions below were used during development of LEAP but other versions will also likely work.

Additional libraries required are easily installable via the pip package manager:

pip install -Iv numpy==1.14.1
pip install -Iv h5py==2.7.1
pip install -Iv tensorflow-gpu==1.6.0
pip install -Iv keras==2.1.4
pip install -Iv clize==4.0.3

You will also need OpenCV 3 with Python bindings. We recommend using skvark's excellent precompiled packages:

pip install -Iv opencv-python==3.4.0.12

Guides coming soon.

1. Cluster sampling: Coming soon.
2. Create skeleton: Call create_skeleton from MATLAB commandline to launch GUI.
3. Label data and train: Call label_joints from MATLAB commandline to launch GUI.
4. Batch estimation: Coming soon.

See leap/training.py and leap/predict_box.py for more info.

Reach out to us via email: Talmo Pereira ([email protected])