Repository for training low-frequency motion control (LFMC) policies for robotic locomotion.

Project website: https://ori-drs.github.io/lfmc/

Deployment (C++): https://github.com/ori-drs/lfmc_cval
Deployment (Python): https://github.com/ori-drs/lfmc_pyval

Preprint: https://arxiv.org/abs/2209.14887

@article{lfmc-locomotion,
  title = {Learning Low-Frequency Motion Control for Robust and Dynamic Robot Locomotion},
  author = {Gangapurwala, Siddhant and Campanaro, Luigi and Havoutis, Ioannis},
  url = {https://arxiv.org/abs/2209.14887},
  publisher = {arXiv},  
  year = {2022},
  doi = {10.48550/ARXIV.2209.14887},
}

The training code is based on Raisim and an adaptation of RaisimGymTorch. The environment is written in C++ while the actual training happens in Python. The current example provided in the repository has been tuned to train a 25 Hz locomotion policy for the ANYmal C robot in roughly 30 minutes on a standard computer with an 8-core Intel i9-9900k CPU @ 3.6 GHz and an NVIDIA RTX 2080 Ti. Additional configurations for training at different frequencies will be added soon.

Another environment, based on Phase Modulating Trajectory Generators (PMTGs), has also been added. This code is based on the work of Lee et al. on Learning Quadrupedal Locomotion over Challenging Terrain and is an adaptation of Mathieu Geisert's implementation of the trajectory generation module.

The training code depnds on Eigen which can be installed in Ubuntu like so:

sudo apt-get install libeigen3-dev

You will also need to have installed Raisim. Please refer to the Raisim documentation for install instructions. Based on the documentation, we will assume that Raisim has been installed in a directory called $LOCAL_INSTALL.

To clone lfmc_gym, use the following command. Note that, this repository depends upon a neural network implementation written in C++ called networks_minimal and is included as a submodule. Ensure you use --recurse-submodule flag while cloning the repository.

git clone --recurse-submodules [email protected]:ori-drs/lfmc_gym.git

Alternatively, you can clone the networks_minimal repository in the dependencies directory.

cd lfmc_gym
git clone [email protected]:gsiddhant/networks_minimal.git dependencies/networks_minimal

We use the header branch of this repository.

cd dependencies/networks_minimal
git checkout header
cd ../..

A setup.py script is handles the necessary dependencies and C++ builds. It is recommended that you use a virtual environment. If python3-venv is not already installed, use the following command.

sudo apt-get install python3-venv

Assuming you are in the project root directory $LFMC_GYM_PATH, create and source a virtual environment.

cd $LFMC_GYM_PATH
python3 -m venv venv
source venv/bin/activate

The relevant build and installs can then be done using

python setup.py develop

The main environment provided is called anymal_velocity_command and is used to train a velocity command tracking locomotion policy. Before you start training, launch the RaisimUnity visualizer and check the Auto-connect option. The training can then be started using

python scripts/anymal_velocity_command/runner.py

After about 5k training iterations, you should observed a good velocity tracking behavior. The checkpoints are stored in the $LFMC_GYM_PATH/data/anymal_velocity_command/<date-time> directory. To test the trained policy, use the provided script.

python scripts/anymal_velocity_command/tester.py

Another environment called anymal_pmtg_velocity_command has now been added. Policies obtained using this approach were observed to be more stable than policies which generate the joint position commands directly. To train policies using this RL environment, follow the method as before except use the scripts provided in $LFMC_GYM_PATH/scripts/anymal_pmtg_velocity_command directory.

└── common                            # Utilities used by modules throughout the project
    ├── paths.py                      # Utility to handle project related paths
└── dependencies                      # Packages required by LFMC-Gym
    ├── actuation_dynamics            # Actuator network for ANYmal C
    ├── networks_minimal              # C++ based implementation of MLP and GRU networks
└── gym_envs                          # RL environments
    ├── anymal_velocity_command       # The ANYmal-Velocity-Command gym environment
        └── Environment.hpp           # Describes the main RL Gym functions
        └── cfg,yaml                  # Environment configuration and training parameters 
    ├── anymal_pmtg_velocity_command  # The ANYmal-PMTG-Velocity-Command gym environment
        └── Environment.hpp           # Describes the main RL Gym functions
        └── cfg,yaml                  # Environment configuration and training parameters 
└── modules                           # Actor-Critic network architectures and helper functions
    ├── ...
└── raisim_gym_torch                  # Python wrapper for Raisim-C++ env and PPO implementation
    ├── ...
└── resources                         # Assets used in the project
    ├── models                        # Robot URDFs
    ├── parameters                    # Neural network parameters (eg. ANYmal C actuator network)
└── scripts                           # Python scripts for training, evaluation and utilities
    ├── anymal_velocity_command       # Training and evaluation scripts for ANYmal-Velocity-Command
        └── runner.py                 # Training script for ANYmal-Velocity-Command
        └── tester.py                 # Evaluation script that executes trained policies 
    ├── utility                       # Scripts to extract model parameters
        └── ...
└── CMakeLists.txt                    # C++ build utility
└── setup.py                          # Executes C++ build and installs LFMC-Gym and dependencies

Siddhant Gangapurwala