Learning in Markov Games with Adaptive Adversaries: Policy Regret, Fundamental Barriers, and Efficient Algorithms

We first show that any learner must incur a linear policy regret against an adaptive opponent who can adapt and remember the learner’s past policies (Theorem 1). When the opponent has a bounded memory span, any learner must require an exponential number of samples $\Omega((SA)^{H}/{\epsilon}^{2})$ to obtain an ${\epsilon}$-suboptimal policy regret, even with the weakest form of memory wherein the opponent is oblivious (Theorem 2). When the memory-bounded opponent’s response is stationary, i.e., the response function does not vary with episodes, learning is still statistically hard when the learner’s policy set is exponentially large, as in this case the policy regret necessarily scales polynomially with the cardinality of the learner’s policy set (Theorem 3).

Motivated by these statistical hardness results, we consider a structural condition on the response of the opponents, which we refer to as consistent behavior, wherein the opponent responds similarly to similar sequences of policies (5). We propose two algorithms OPO-OMLE (Algorithm 1) and APE-OVE (Algorithm 3) that obtain $\sqrt{T}$ policy regret against $m$-memory bounded, stationary, and consistent adversaries, for $m=1$ and $m\geq 1$, respectively.

• •

  For memory length $m=1$: We show that OPO-OMLE obtains a policy regret upper bound of $\tilde{{\mathcal{O}}}(H^{3}S^{2}AB+\sqrt{H^{5}SA^{2}BT})$, when the learner’s policy set is the set of all deterministic Markov policies, where $H$ is the episode length, $S$ is the number of states, $A$ and $B$ are the numbers of actions for the learner and the opponent, respectively, and $T$ is the number of episodes.

• •

  For general memory length $m\geq 1$: We show that APE-OVE obtains a policy regret upper bound of $\tilde{{\mathcal{O}}}\left((m-1)H^{2}SAB+\sqrt{H^{3}SAB}(SAB(H+\sqrt{S})+H^{2})\sqrt{\frac{T}{d^{*}}}\right)$, where $d^{*}$ is an instance-dependent quantity that features the minimum positive visitation probability.

We provide a summary of our main results in Table 1.

Learning problems in Markov games have been studied extensively in the MARL literature. Most existing works focus on learning Nash equilibria either with known dynamics or infinite data (Littman, 1994; Hu and Wellman, 2003; Hansen et al., 2013; Wei et al., 2020), or otherwise in a self-play setting wherein we control all the players (Wei et al., 2017; Bai et al., 2020; Bai and Jin, 2020; Xie et al., 2020; Liu et al., 2021), or in an online setting wherein we control one player to learn against other potentially adversarial players (Brafman and Tennenholtz, 2002; Wei et al., 2020; Tian et al., 2021; Jin et al., 2022). Other related work focuses on exploiting sub-optimal opponents via no-external regret learning (Liu et al., 2022) and studying Stackelberg equilibria in two-player general-sum turn-based MGs, wherein only one player is allowed to take actions in each state (Ramponi and Restelli, 2022).

Policy regret minimization has been studied mostly in online (bandit) learning problems. It was first studied in a full information setting (Merhav et al., 2002) and extended to the bandit setting and more powerful competitor classes using swap regret and $\Phi$-regret (Arora et al., 2012). A lower bound of $T^{2/3}$ on policy regret in a bandit setting was provided by Dekel et al. (2014) and was later extended to action space with metric (Koren et al., 2017a, b). A long line of works studies (complete) policy regret in “tallying” bandits, wherein an action’s loss is a function of the number of the action’s pulls in the previous $m$ rounds (Heidari et al., 2016; Levine et al., 2017; Seznec et al., 2019; Lindner et al., 2021; Awasthi et al., 2022; Malik et al., 2022, 2023).

Beyond online (bandit) learning, policy regret has been studied in several more challenging settings. In Arora et al. (2018) authors study the notion of policy equilibrium in repeated games (Markov games with $H=S=1$) when agents follow no-policy regret algorithms. A more complete characterization of the learnability in online learning with dynamics, where the loss function additionally depends on time-evolving states, was given in Bhatia and Sridharan (2020). Finally, in Dinh et al. (2023), authors study policy regret in online MDP, where an adversary who follows a no-external regret algorithm generates the loss functions, which effectively alleviates policy regret minimization to the standard external regret minimization in online MDPs.

In this paper, we use the framework of Markov Games to study an interactive multi-agent decision-making and learning environment (Shapley, 1953). Markov games extend Markov decision processes (MDPs) to multiplayer scenarios, where each agent’s action affects not only the environment but also the subsequent state of the game and the actions of other agents. Formally, a standard two-player Markov Game (MG) is specified by a tuple $M=({\mathcal{S}},{\mathcal{A}},{\mathcal{B}},H,P,r)$. Here, ${\mathcal{S}}$ denotes the state space with cardinality $|{\mathcal{S}}|=S$, ${\mathcal{A}}$ is the action space of the first player (called learner) with cardinality $|{\mathcal{A}}|=A$, ${\mathcal{B}}$ is the action space of the second player (referred to as an opponent or an adversary) with cardinality $|{\mathcal{B}}|=B$, $H\in{\mathbb{N}}$ is the time horizon for each game. $P=\{P_{1},\ldots,P_{H}\}$ are the transition kernels with each $P_{h}:{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{B}}\rightarrow\Delta(S)$ specifying the probability of transitioning to the next state given the current state, learner’s action, and adversary’s action ($\Delta({\mathcal{S}})$ denotes the set of all probability distributions over ${\mathcal{S}}$). Finally, $r=\{r_{1},\ldots,r_{H}\}$ are the (expected) reward functions with each $r_{h}:{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{B}}\rightarrow[0,1]$. For simplicity, we assume the learner knows the reward function.¹¹1Our results immediately generalize to unknown reward functions, as learning the transitions is more difficult than learning the reward functions in tabular MGs.

Each episode begins in a fixed initial state $s_{1}$. At step $h\in[H]$, the learner observes the state $s_{h}$ and picks her action $a_{h}\in{\mathcal{A}}$ while the opponent/adversary picks an action $b_{h}\in{\mathcal{B}}$. As a result, the learner observes $b_{h}$, receives reward $r_{h}(s_{h},a_{h},b_{h})$ and the environment transitions to $s_{h+1}\sim P_{h}(\cdot|s_{h},a_{h},b_{h})$. The episode terminates after $H$ steps.

A learner’s policy (also referred to as strategy) is any tuple $\pi=\{\pi_{h}\}_{h\in[H]}$ where $\pi_{h}:({\mathcal{S}}\times{\mathcal{A}})^{h-1}\times{\mathcal{S}}\rightarrow\Delta({\mathcal{A}})$. A policy $\pi=\{\pi_{h}\}_{h\in[H]}$ is said be Markovian if for every $h\in[H],\pi_{h}:{\mathcal{S}}\rightarrow\Delta({\mathcal{A}})$. Similarly, an adversary’s policy is any tuple $\mu=\{\mu_{h}\}_{h\in[H]}$ where $\mu_{h}:({\mathcal{S}}\times{\mathcal{B}})^{h-1}\times{\mathcal{S}}\rightarrow\Delta({\mathcal{B}})$. $\mu$ is said to be Markovian if for every $h$, $\mu_{h}:{\mathcal{S}}\rightarrow\Delta({\mathcal{B}})$. For simplicity, we will focus only on Markov policies for both the learner and the adversary in this paper. Let $\Pi$ (respectively, $\Psi$) be the set of all feasible policies of the learner (respectively, the adversary). The value of a policy tuple $(\pi,\mu)\in\Pi\times\Psi$ at step $h$ in state $s$, denoted by $V_{h}^{\pi,\mu}(s)$ is the expected accumulated reward starting in state $s$ from step $h$, if the learner and the adversary follow $\pi$ and $\mu$ respectively, i.e., $V_{h}^{\pi,\mu}(s):={\mathbb{E}}_{\pi,\mu}[\sum_{l=h}^{H}r_{l}(s_{l},a_{l},b_{l})|s_{h}=s]$, where the expectation is with respect to the trajectory $(s_{1},a_{1},b_{1},r_{1},\ldots,s_{H},a_{H},b_{H},r_{H})$ distributed according to $P$, $\pi$, and $\mu$. We also denote the action-value function $Q^{\pi,\mu}_{h}(s,a,b):={\mathbb{E}}_{\pi,\mu}[\sum_{l=h}^{H}r_{l}(s_{l},a_{l},b_{l})|(s_{h},a_{h},b_{h})=(s,a,b)]$. Given a $V:{\mathcal{S}}\rightarrow{\mathbb{R}}$, we write $P_{h}V(s,a,b):={\mathbb{E}}_{s^{\prime}\sim P_{h}(\cdot|s,a,b)}[V(s^{\prime})]$. For any $u:{\mathcal{S}}\rightarrow\Delta({\mathcal{A}})$, $v\!:\!{\mathcal{S}}\rightarrow\Delta({\mathcal{B}})$, $Q\!:\!{\mathcal{S}}\!\times\!{\mathcal{A}}\!\times\!{\mathcal{B}}\rightarrow{\mathbb{R}}$, denote $Q(s,u,v):={\mathbb{E}}_{a\sim u(\cdot|s),b\sim v(\cdot|s)}[Q(s,a,b)]$ for any $s\in{\mathcal{S}}$.

We allow the adversary to be adaptive, i.e., the adversary can choose their policy in episode $t$ based on the learner’s policies on episodes $1,\ldots,t$. We assume that the adversary is deterministic and has unlimited computational power, i.e., the adversary can plan, in advance, using as much computation as needed, as to how they would react in each episode to any sequence of policies. Formally, the adversary defines in advance a sequence of deterministic functions $\{f_{t}\}_{t\in{\mathbb{N}}^{*}}$, where $f_{t}:\Pi^{t}\rightarrow\Psi$. The input to each response function $f_{t}$ is an entire history of the learner’s policies, including her policy in episode $t$. Therefore, if the learner follows policies $\pi^{1},\ldots,\pi^{t}$, the adversary responds with policy $f_{t}(\pi^{1},\ldots,\pi^{t})\in\Psi$ in episode $t$. Since the response function $f_{t}$ depends on the learner’s policy at round $t$, our setup is essentially a principal-follower model, akin to Stackelberg games (Letchford et al., 2009; Blum et al., 2014) and mechanism design for learning agents (Braverman et al., 2019). In this context, the principal agent (mechanism designer or learner) publicly declares a strategy before committing to it, allowing the followers to subsequently choose their strategies based on their understanding of the principal’s decisions.

We evaluate the learner’s performance using the notion of policy regret (Merhav et al., 2002; Arora et al., 2012), which compares the return on the first $T$ episodes to the return of the best fixed sequence of policy in hindsight. Formally, the learner’s policy regret after $T$ episodes is defined as

Policy regret has been studied in online (bandit) learning (Merhav et al., 2002; Arora et al., 2012) and repeated games (Arora et al., 2018), yet, to the best of our knowledge, it has never been studied in Markov games. Policy regret differs from the more common definition of external regret defined as $R(T)=\sup_{\pi\in\Pi}\sum_{t=1}^{T}V_{1}^{\pi,f_{t}(\pi^{1},\ldots,\pi^{t})}(s_{1})-V_{1}^{\pi^{t},f_{t}(\pi^{1},\ldots,\pi^{t})}(s_{1})$, which is used in (Liu et al., 2022). However, external regret is inadequate for measuring the learner’s performance against an adaptive adversary. Indeed, when the adversary is adaptive, the quantity $V_{1}^{\pi,f_{t}(\pi^{1},\ldots,\pi^{t})}$ is hardly interpretable anymore – see (Arora et al., 2012) for a more detailed discussion.

As a warm-up, we show in the following example that, policy regret minimization generalizes the standard Nash equilibrium learning problem in zero-sum two-player Markov games.

We write $f\lesssim g$ to mean $f={\mathcal{O}}(g)$. We use $c$ to represent an absolute constant that can have different values in different appearances.

While seemingly restrictive, policy regret minimization with $m$-memory bounded and stationary adversaries already subsumes the problem of learning Stackelberg equilibrium (Von Stackelberg, 2010) in general-sum Markov games (Ramponi and Restelli, 2022).³³3Ramponi and Restelli (2022) consider a more restrictive setting of turn-based Markov games, wherein at each state only one player is allowed to take actions. In addition, they require the opponents to respond with only deterministic policies. In general-sum Markov games, the adversary (“follower”) aims at maximizing his own reward function given any policy of the learner (“leader”). That is, the adversary is $1$-memory bounded, and the response function $f:\Pi\rightarrow\Psi$ corresponds to a function that selects the best response policy to any given policy of the learner. The benchmark $\max_{\pi\in\Pi}V_{1}^{\pi,f(\pi)}$ in policy regret then becomes the Stackelberg equilibrium.

Is sample-efficient learning possible against $m$-memory bounded and stationary adversaries? One can notice an immediate approach to learning against a $1$-memory bounded and stationary adversaries is to simply view the problem as a $|\Pi|$-armed bandit problem and apply any state-of-the-art bandit algorithm (Audibert and Bubeck, 2009) to obtain ${\textrm{PR}}(T)={\mathcal{O}}(H\sqrt{T|\Pi|})$. However, scaling polynomially with the learner’s policy class is not desirable when the class is exponentially large (e.g., when the learner’s policy class is the set of all deterministic policies, then $|\Pi|=\Theta(A^{HS})$). And in fact, we cannot avoid polynomial scaling with the cardinality of the learner’s policy class in general.

Note that the lower bound applies to $m=1$, and, therefore, to any $m\geq 1$. Proof in Section A.3.