Implementations Of Counterfactual Regret Minimization In Its Many Shapes & Colors

CFRainbow provides implementations of the basic CFR algorithm and some of the improved variants of CFR for computing Nash Equilibria in the 2-player 0-sum case as well as Correlated Equilibria in the general-sum case.

The pacakge is loosely built for modularity and general applicability by building on the Openspiel framework by Deepmind. CFRainbow is not built for performance and will not scale well to large game implementations. Most algorithms are implemented with basic python data structures.

The package is in the early WIP phase.

The easiest way to use CFRainbow is to import the desired algorithm from the package, a regret minimizer, and the run method with a pyspiel game object or its OpenSpiel name as input. For example, to use the Vanilla CFR algorithm:

import cfrainbow
from cfrainbow import cfr, rm

cfrainbow.run(
  cfr.VanillaCFR,
  n_iter=1000,
  game="kuhn_poker",
  regret_minimizer=rm.RegretMatcher
)

This will run the algorithm for the number of iterations given and compute the exploitability.

You can also use the solver object directly and call iterate on it after instantiating it with the correct arguments. For example:

from cfrainbow import cfr, rm
from cfrainbow.utils import load_game, KuhnPolicyPrinter, normalize_policy_profile

root_state = load_game("kuhn_poker").new_initial_states()

solver = cfr.VanillaCFR(
    root_state,
    regret_minimizer_type=rm.RegretMatcher,
    alternating=True,  # whether to do alternating or simultaneous updates
)
# iterate for a given number of iterations.
for i in range(1000):
    solver.iterate()

avg_policy = normalize_policy_profile(solver.average_policy())

print(
    KuhnPolicyPrinter(digits=2).print_profile(avg_policy)
)

Output (Infostate --> Action Policy):

PLAYER 0:
P1: Jack  | P2:   ?   | cb   --> ['check:  1.00', 'bet:  0.00']
P1: Jack  | P2:   ?          --> ['check:  0.81', 'bet:  0.19']
P1: Queen | P2:   ?   | cb   --> ['check:  0.48', 'bet:  0.52']
P1: Queen | P2:   ?          --> ['check:  0.99', 'bet:  0.01']
P1: King  | P2:   ?   | cb   --> ['check:  0.00', 'bet:  1.00']
P1: King  | P2:   ?          --> ['check:  0.42', 'bet:  0.58']
PLAYER 1:
P1:   ?   | P2: Queen | c    --> ['check:  0.99', 'bet:  0.01']
P1:   ?   | P2: Queen | b    --> ['check:  0.66', 'bet:  0.34']
P1:   ?   | P2: King  | c    --> ['check:  0.00', 'bet:  1.00']
P1:   ?   | P2: King  | b    --> ['check:  0.00', 'bet:  1.00']
P1:   ?   | P2: Jack  | c    --> ['check:  0.67', 'bet:  0.33']
P1:   ?   | P2: Jack  | b    --> ['check:  1.00', 'bet:  0.00']

Note that in alternating updates each iteration is a single player's policy update. In simultaneous updates both players' policy updates constitute one iteration.

The following list shows the available algorithms that have been implemented (✔️), those that are still work in progress (🔨👷‍♂️), and those that are planned to be implemented in the future (📅):

Pip

To install CFRainbow from PyPi using pip, run the following command:

pip install cfrainbow

Poetry

If you're using Poetry to manage your Python packages, you can add CFRainbow to your project by adding the following to your pyproject.toml file:

[tool.poetry.dependencies]
cfrainbow = "*"

Then run:

poetry install

to install your own package. Poetry will then install cfrainbow as dependency alongside your own package.

To install CFRainbow from master, please follow these steps:

git clone https://github.com/maichmueller/cfrainbow.git
cd cfrainbow

and install the package with pip

pip install .

or poetry

poetry install

use the option --no-dev to ensure a non-editable installation.