An easy-to-use framework for reinforcement learning in PyTorch, accelerated with Lightning Fabric.
The algorithms sheeped by sheeprl out-of-the-box are:

and more are coming soon! Open a PR if you have any particular request 🐑

The actions supported by sheeprl agents are:

The environments supported by sheeprl are:

We want to provide a framework for RL algorithms that is at the same time simple and scalable thanks to Lightning Fabric.

Moreover, in many RL repositories, the RL algorithm is tightly coupled with the environment, making it harder to extend them beyond the gym interface. We want to provide a framework that allows to easily decouple the RL algorithm from the environment, so that it can be used with any environment.

Two options exist for using SheepRL. One can either clone the repo and install the local version, or one can pip install the framework using the GitHub clone URL. Instructions for both methods are shown below.

git clone https://github.com/Eclectic-Sheep/sheeprl.git
cd sheeprl

pip install .

# create a virtual environment
python3 -m venv .venv
# activate the environment
source .venv/bin/activate
# if you do not wish to install extras such as mujuco, atari do
pip install "sheeprl @ git+https://github.com/Eclectic-Sheep/sheeprl.git"
# or, to install with atari and mujuco environment support, do
pip install "sheeprl[atari,mujoco,dev]  @ git+https://github.com/Eclectic-Sheep/sheeprl.git"
# or, to install with minedojo environment support, do
pip install "sheeprl[minedojo,dev]  @ git+https://github.com/Eclectic-Sheep/sheeprl.git"
# or, to install with minerl environment support, do
pip install "sheeprl[minerl,dev]  @ git+https://github.com/Eclectic-Sheep/sheeprl.git"
# or, to install with diambra environment support, do
pip install "sheeprl[diambra,dev]  @ git+https://github.com/Eclectic-Sheep/sheeprl.git"
# or, to install all extras, do
pip install "sheeprl[atari,mujoco,miedojo,dev,test]  @ git+https://github.com/Eclectic-Sheep/sheeprl.git"

# if needed install homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# then, do
brew install swig
# then attempt to pip install with the prefered method, such as
pip install "sheeprl[atari,mujoco,dev,test] @ git+https://github.com/Eclectic-Sheep/sheeprl.git"

Now you can use one of the already available algorithms, or create your own.

For example, to train a PPO agent on the CartPole environment with only vector-like observations, just run

python sheeprl.py exp=ppo env=gym env.id=CartPole-v1

You check all the available algorithms with

python sheeprl/available_agents.py

That's all it takes to train an agent with SheepRL! 🎉

Note

Before you start using the SheepRL framework, it is highly recommended that you read the following instructional documents:

1. How to run experiments
2. How to modify the default configs
3. How to work with steps
4. How to select observations

Moreover, there are other useful documents in the howto folder, which containes some guidance on how to properly use the framework.

Once you trained an agent, a new folder called logs will be created, containing the logs of the training. You can visualize them with TensorBoard:

tensorboard --logdir logs

What you run is the PPO algorithm with the default configuration. But you can also change the configuration by passing arguments to the script.

For example, in the default configuration, the number of parallel environments is 4. Let's try to change it to 8 by passing the --num_envs argument:

python sheeprl.py exp=ppo env=gym env.id=CartPole-v1 env.num_envs=8

All the available arguments, with their descriptions, are listed in the sheeprl/config directory. You can find more information about the hierarchy of configs here.

To run the algorithm with Lightning Fabric, you need to specify the Fabric parameters through the CLI. For example, to run the PPO algorithm with 4 parallel environments on 2 nodes, you can run:

python sheeprl.py fabric.accelerator=cpu fabric.strategy=ddp fabric.devices=2 exp=ppo env=gym env.id=CartPole-v1

You can check the available parameters for Lightning Fabric here.

The repository is structured as follows:

• algos: contains the implementations of the algorithms. Each algorithm is in a separate folder, and (possibly) contains the following files:

  • <algorithm>.py: contains the implementation of the algorithm.
  • <algorithm>_decoupled.py: contains the implementation of the decoupled version of the algorithm, if present.
  • agent: optional, contains the implementation of the agent.
  • loss.py: contains the implementation of the loss functions of the algorithm.
  • utils.py: contains utility functions for the algorithm.

• configs: contains the default configs of the algorithms.

• data: contains the implementation of the data buffers.

• envs: contains the implementation of the environment wrappers.

• models: contains the implementation of the some standard models (building blocks), like the multi-layer perceptron (MLP) or a simple convolutional network (NatureCNN)

• utils: contains utility functions for the framework.

In the coupled version of an algorithm, the agent interacts with the environment and executes the training loop.

In the decoupled version, a process is responsible only for interacting with the environment, and all the other processes are responsible for executing the training loop. The two processes communicate through distributed collectives, adopting the abstraction provided by Fabric's TorchCollective.

The algorithm is implemented in the <algorithm>.py file.

There are 2 functions inside this script:

• main(): initializes all the components of the algorithm, and executes the interactions with the environment. Once enough data is collected, the training loop is executed by calling the train() function.
• train(): executes the training loop. It samples a batch of data from the buffer, compute the loss and updates the parameters of the agent.

The decoupled version of an algorithm is implemented in the <algorithm>_decoupled.py file.

There are 3 functions inside this script:

• main(): initializes all the components of the algorithm, the collectives for the communication between the player and the trainers and calls the player() and trainer() functions.
• player(): executes the interactions with the environment. It samples an action from the policy network, executes it in the environment, and stores the transition in the buffer. After a predefined number of iteractions with the environment, the player randomly splits the collected data in almost-equal chunks and send them separately to the trainers. It then waits for the trainers to finish the agent update.
• trainer(): executes the training loop. It receives a chunk of data from the player, compute the loss and updates the parameters of the agent. After the agent has been updated, the first of the trainers sends back the updated agent weights to the player, which can interact again with the environment.

You can check inside the folder of each algorithm the README.md file for the details about the implementation.

All algorithms are kept as simple as possible, in a CleanRL fashion. But to allow for more flexibility and also more clarity, we tried to abstract away anything that is not strictly related with the training loop of the algorithm.

For example, we decided to create a models folder with an already-made models that can be composed to create the model of the agent.

For each algorithm, losses are kept in a separate module, so that their implementation is clear and can be easily utilized also for the decoupled or the recurrent version of the algorithm.

For the buffer implementation, we choose to use a wrapper around a TensorDict.

TensorDict comes handy since we can easily add custom fields to the buffer as if we are working with dictionaries, but we can also easily perform operations on them as if we are working with tensors.

This flexibility makes it very simple to implement, with the classes ReplayBuffer, SequentialReplayBuffer, EpisodeBuffer, and AsyncReplayBuffer, all the buffers needed for on-policy and off-policy algorithms.

The tensor's shape in the TensorDict are (T, B, *), where T is the number of timesteps, B is the number of parallel environments, and * is the shape of the data.

For the ReplayBuffer to be used as a RolloutBuffer, the proper buffer_size must be specified. For example, for PPO, the buffer_size must be [T, B], where T is the number of timesteps and B is the number of parallel environments.

The best way to contribute is by opening an issue to discuss a new feature or a bug, or by opening a PR to fix a bug or to add a new feature.

You can contact us for any further questions or discussions:

• Federico Belotti: [email protected]
• Davide Angioni: [email protected]
• Refik Can Malli: [email protected]
• Michele Milesi: [email protected]