This repository contains my own implementations of different actor-critic methods. It uses Weights&Biases for training statistics and visualizations. For long experiments, the statistics file saved can get very big in memory size.

Trained agent playing CartPole	Trained agent playing FlappyBird

Available implementations:

• REINFORCE with Critic as Baseline
• Batch Actor-Critic
• Online Actor-Critic
• Advantage Actor-Critic (A2C)

There is an interface to add environments to train the agents. At this moment it has:

• CartPole-v0
• FlappyBird

Install all needed packages with:

pip install -r requirements.txt

To train a new agent to play one of the available environments, use the script train_agent.py. The experiment configuration is passed as a .json file. The configurations folder has files with the best set of hyperparameters I found for each agent and environment.

Here is an example to train a Batch Actor-Critic agent on CartPole with default configurations:

python train_agent.py --config_file configurations/cart_pole_batchAC.json

To use new configurations just change the values of the file you are using, or create a copy with different values and pass it to the script. The trained agent files are saved by default in an experiments folder inside this project.

To see all the script options use:

python train_agent.py -h

To test a trained agent on the environment, use the script test_agent.py. You can watch the agent playing with the --render_games flag.

Here is an example to test and watch a trained agent for CartPole located in experiments/cart_pole_batchAC for 10 episodes:

python test_agent.py --experiment_dir experiments/cart_pole_batchAC --episodes 10 --render_games

By default, the agents stores the state of itself 20 times during the complete training process. The script animated_progress.py shows the agent playing a game one time for each saved state. It lets you visualize how the agent makes progress in trying to solve the current task.

Here is an example to visualize the progress of an agent for CartPole located in experiments/cart_pole_batchAC:

python animated_progress.py --experiment_dir experiments/cart_pole_batchAC

Here are my experiments best results for each environment and agent type. The experiment duration is on my MSI GeForce RTX 2060 SUPER, for a time reference frame.

The mean test reward is running the agent for 100 episodes. For FlappyBird a perfect score is 10, for CartPole is 200.

Here are all the experiments for each environment. You can download and visualize them using the scripts:

• CartPole
• FlappyBird

• https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf
• https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic
• https://incompleteideas.net/sutton/book/RLbook2018.pdf
• https://github.com/ntasfi/PyGame-Learning-Environment
• https://gym.openai.com/