JP4586129B2 - Controller, control method and control program - Google Patents

Description

  The present invention relates to the configuration of a controller, a control method, and a control program for controlling a controlled object by a policy gradient method.

  The control problem formulated as a “Markov decision process” is an important technology that has a wide range of applications as an autonomous control problem for robots, plants, mobile machines (trains, cars, etc.).

  There is so-called “reinforcement learning” as a conventional technique related to optimal control for a Markov decision process.

  “Reinforcement learning” is a behavioral rule called “policy” that maximizes the cumulative amount of reward that is obtained through trial and error through interaction with the environment. It is a theoretical framework for learning. This learning method attracts attention from various fields in that it requires little a priori knowledge about the environment and the agent itself.

  Reinforcement learning can be roughly classified into two. The value function is used to express the policy indirectly, and the value function is updated to update the policy, and the value function is updated, and the policy is updated according to the objective function gradient. “Policy update method (policy gradient method)”.

  The policy gradient method is particularly attracting attention because it can acquire a stochastic policy by including a parameter for controlling the randomness of the action in the policy parameter, and has high applicability to a continuous system. However, in general, when applied to a real task, the time to acquire an appropriate behavioral rule may be unrealistic. Therefore, active research has been conducted to shorten the learning time by using auxiliary mechanisms such as simultaneous use of multiple learners, use of models, use of teaching signals, etc., and results have been remarkable.

  Here, the policy gradient reinforcement learning method (PGRL) is a method of reinforcement learning (RL: Reinforcemnt Learning) for maximizing the average reward by improving the policy parameter by using partial differentiation of the average reward for the policy parameter. It is a general algorithm. Here, the partial differential of the average reward is called a policy gradient (PG). Since the conventional policy gradient reinforcement learning method algorithm (PG algorithm) [Non-Patent Document 1, Non-Patent Document 2] is difficult to calculate the partial derivative of the steady distribution of states, Ignoring policy gradient terms that depend on change. These are called logarithmic partial derivatives of stationary distributions -LSDG ((Log) Stationary Distribution Gradients)-. The bias due to such omission decreases as the so-called discount rate γ approaches 1, but on the other hand, the variance of the estimated partial differential increases. Such a trade-off has made it difficult to actually find an appropriate γ. Mixing the Markov chain time is a measure for determining an appropriate γ [Non-Patent Document 1, Non-Patent Document 3]. However, the mixing of time depends on the strategy and is therefore generally difficult to predict before learning is complete.

  Further, the average reward PG algorithm [Non-Patent Document 4, Non-Patent Document 5] does not use a discount rate by introducing a differential cost function as a solution of the Poisson equation. However, it has been suggested that there is no theoretical difference between the performance of a normal PG with a discount rate close to 1 and the average reward PG [6].

  Therefore, there is a need for a learning method that solves the trade-off of increasing variance when the bias of estimated policy parameters as described above is reduced.

Hereinafter, prior art documents related to the policy gradient learning method to be cited in the text are listed below.
Baxter, J. and P. Bartlett (2001) “Infinite-Horizon Policy-Gradient Estimation,” Journal of Artificial Intelligence Research, Vol. 15, pp. 319-350. H. Kimura and S. Kobayashi.An analysis of actor / critic algorithms using eligibility traces: Reinforcement learning with imperfect value function.In International Conference on Machine Learning, 1998. S. Kakade. Optimizing average reward using discounted rewards.In Annual Conference on Computational Learning Theory, volume 14.MIT Press, 2001. JN Tsitsiklis and B. Van Roy.Average cost temporal-difference learning.Automatica, 35 (11): 1799-1808, 1999. VS Konda and JN Tsitsiklis.On actor-critic algorithms.SIAM Journal on Control and Optimization, 42 (4): 1143-1166, 2001. ] JN Tsitsiklis and B. Van Roy. On average versus discounted reward temporal-difference learning. Machine Learning, 49 (2): 179-191, 2002. PW Glynn. Likelihood ratio gradient estimation for stochastic systems.Communications of the ACM, 33 (10): 75-84, 1991. RY Rubinstein. How to optimize discrete-event system from a single sample path by the score function method.Annals of Operations Research, 27 (1): 175-212, 1991. AY Ng, R. Parr, and D. Koller.Policy search via density estimation.In Advances in Neural Information Processing Systems.MIT Press, 2000. DP Bertsekas.Dynamic Programming and Optimal Control, Volumes 1 and 2.Athena Scientific, 1995. RS Sutton and AG Barto. Reinforcement Learning. MIT Press, 1998. RS Sutton. Learning to predict by the methods of temporal differences.Machine Learning, 3: 9-44, 1988. SJ Bradtke and AG Barto.Linear least-squares algorithms for temporal difference learning.Machine Learning, 22 (1-3): 33-57, 1996. JA Boyan. Least-squares temporal difference learning.Machine Learning, 49 (2-3): 233-246, 2002. RB Schinazi. Classical and Spatial Stochastic Processes. Birkhauser, 1999. J. Peng and RJ Williams. Incremental multi-step Q-learning. Machine Learning, 22: 283-290, 1996. P. Young.Recursive Esimation and Time-series Analysis.Springer-Verlag, 1984. DP Bertsekas and JN Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.

  There are already two methods for evaluating the gradient of the (steady) state distribution, but these are different from the method of the present invention and have the following problems. The first one is called “likelihood ratio gradient method” or “score function method” [Non-Patent Document 7, Non-Patent Document 8], and has a problem that can be applied only to the reproduction process [Non-Patent Document 1]. . The other method disclosed in Non-Patent Document 9 is executed not through direct evaluation of state distribution but through evaluation of state distribution with density propagation. Therefore, these methods require knowledge of which state the agent is in, whereas in the method of the present invention described later, it is only necessary to observe a feature vector of a state including noise.

  Therefore, the present invention has been made to solve the above-described problems, and the object thereof is a policy gradient learning method that solves the trade-off that the variance increases when the policy parameter bias is reduced. The present invention provides a controller, a control method, or a control program using a computer.

  Another object of the present invention is to provide a controller, control method or control program using a versatile policy gradient learning method.

In order to achieve such an object, the controller of the present invention is expressed stochastically as a control law for the state of the system when the time evolution of the target system is described as a forward Markov decision process. that measures a controller for reinforcement learning by policy gradient by the observation of the state of the system, based on policy, and a control signal generating means for generating a control signal for controlling the system, the state of the system The policy is defined by the state quantity detection means to be observed, the reward value acquisition means for acquiring a reward value that depends on a predetermined relationship between the state and the control signal, and the policy parameter that is a parameter that defines the stochastic policy. Rutoki, on the basis of the state quantity and the control signal in each time step is the partial derivative of the logarithm of the measures parameter of the normal distribution of the distribution state of the system By estimating the number stationary distribution partial differential, the policy gradient estimation means for estimating a gradient of measures, based on the estimation result of the compensation value and a policy gradient estimation means, the average compensation estimated using the log-normal distribution partial derivative the square measures parameters in the direction of the partial differential be to small changes, and a measure updating means for updating the policy.

Preferably, the policy gradient estimation means is a constraint condition derived by a condition that the sum of the state distribution states is constant, and the expected value for the forward Markov chain of log-steady distribution partial differentiation is 0 Under the constraint that there is a logarithmic steady distribution partial derivative is estimated by TD learning for the backward Markov chain .
Preferably, in TD learning, i) the sum of the observed value one step before the logarithmic partial differentiation of the policy in the backward Markov chain, the logarithmic steady distribution partial derivative one step before, and the log steady distribution bias of the current state When the difference from the derivative is δ, the sum of the expected value of the forward square Markov chain with the square of δ and the expected value square of the forward Markov chain of the logarithmic steady distribution partial differential is minimized. By doing so, the logarithmic steady distribution partial differential is estimated.

According to another aspect of the present invention, when the time evolution of the target system is described as a forward Markov decision process , a policy expressed stochastically, which is a control law for the system state, A control method for reinforcement learning by means of a policy gradient method by observation, a control signal generation step for generating a control signal for controlling the system based on the policy, a state quantity detection step for observing the state quantity of the system, When a policy is defined by a reward value acquisition step that acquires a reward value that depends on a state and a control signal in a predetermined relationship, and a policy parameter that is a parameter that defines a probabilistic policy, the state at each time step based on the amount and the control signal is a partial differential of the logarithm of the measures parameter of the normal distribution of the distribution state of the system log steady By estimating the fabric partial differential, gradient of the policy gradient estimating step for estimating a gradient of measures, based on the estimation result of the compensation value and a policy gradient estimation means, average earnings expressed on the basis of the log-normal distribution partial derivative by the direction update the square measures parameters, and a policy update step of updating the policy.

According to still another aspect of the present invention, when the time evolution of the target system is described as a forward Markov decision process , a policy expressed stochastically, which is a control law for the system state, is represented as a system state quantity. a program for executing a control method for enhancing learning in a computer by a policy gradient method by the observation, based on the policy, and a control signal generating step of generating a control signal for controlling the system, the state of the system A policy is defined by a state quantity detection step for observing a state, a reward value acquisition step for acquiring a reward value that depends on a predetermined relationship between the state and the control signal, and a policy parameter that is a parameter for defining a probabilistic expression when it is, on the basis of the state quantity and the control signal at each time step, a pair of stationary distribution of the distribution of the state of the system By estimating a partial differential of measures parameters logarithmic normal distribution partial derivative, and policy gradient estimating step for estimating a gradient of measures, based on the estimation result of the compensation value and a policy gradient estimation step, logarithmic normal distribution by updating the square measures parameters the direction of the gradient of the average earnings expressed on the basis of partial differential, and a policy update step of updating the strategy to perform the control method on a computer.

  The outline of the configuration of the following description will be explained. In (1. Overview of the present invention), the overall configuration of the present invention will be explained. In (2. Premise: Policy gradient reinforcement learning), the conventional PG method is reviewed. The purpose of evaluating the LSDG of the present invention will be outlined. (3. Estimation of partial differential of logarithm of stationary distribution) describes an LSDG (λ) algorithm based on the least square TD method based on the inverse Markov chain method. (4. Policy update by LSDG estimation) describes the LSDG (λ) -PG algorithm. This is a method that uses LSDG (λ) and does not use a discount rate γ. In (5. Result of numerical calculation), in order to confirm the performance of the proposed method, the calculation result in a simple Markov decision process (MDP) is shown.

(1. Overview of the present invention)
As will be described later, the present invention proposes a new policy gradient method framework in which LSDG as a gradient of a steady distribution is derived by a backward Markov chain method and a TD (temporal difference) learning algorithm. In this framework, the partial differentiation of the average reward does not depend on the discount rate γ, and by setting γ = 0, it is not necessary to learn the value function.

Embodiments of the present invention will be described below with reference to the drawings.
As will be apparent from the following description, the present invention has a wide range of applications as control problems for robots, plants, mobile machines (trains, automobiles), and the like.

  However, in the following, as a specific application example of the present invention, a description will be given on the assumption that a particularly simple automatic robot control problem is targeted. The numerical calculation results show a comparison with a simpler model. However, the present invention is not limited to such an application, and more generally can be applied to control of the target system when the time development of the target system is complicated. Examples of such are giant plants (blast furnaces, nuclear power plants), multi-link robots (humanoid robots), non-holonomic systems (space stations), and the flow of people in subway platforms. All of these are difficult to control by the classical control method and are important control objects.

(1. System configuration of the present invention)
FIG. 1 is a conceptual diagram showing an example of a system 1000 using a controller to which a control method and a control program of the present invention are applied.

  Referring to FIG. 1, system 1000 includes controlled device 200 to be controlled and a computer 100 for giving a control signal to controlled device 200.

  Referring to FIG. 1, this computer 100 reads / writes information from / to a CD-ROM drive 108 and a flexible disk (FD) 116 for reading information on a CD-ROM (Compact Disc Read-Only Memory). A computer main body 102 having the FD drive 106, a display 104 as a display device connected to the computer main body 102, and a keyboard 110 and a mouse 112 as input devices also connected to the computer main body 102.

FIG. 2 is a block diagram showing the configuration of the computer 100. As shown in FIG.
As shown in FIG. 2, in addition to the CD-ROM drive 108 and the FD drive 106, the computer main body 102 constituting the computer 100 includes a CPU (Central Processing Unit) 120 connected to the bus BS, and a ROM ( It includes a memory 122 including a read only memory (RAM) and a random access memory (RAM), a direct access memory device, for example, a hard disk 124, and a communication interface 128 for exchanging data with the controlled device 200. A CD-ROM 118 is attached to the CD-ROM drive 108. An FD 116 is attached to the FD drive 106.

  From the controlled device 200, information on parameters (state quantities) indicating the state of the controlled device 200 with respect to the computer 100, for example, information on the position, speed, acceleration, angle, angular velocity, etc. of the movable part of the controlled device 200 Is given. On the other hand, control information for controlling these state quantities is given as a control signal from the computer 100 to the controlled device 200.

  The CD-ROM 118 may be another medium, such as a DVD-ROM (Digital Versatile Disc) or a memory card, as long as it can record information such as a program installed in the computer main body. In this case, the computer main body 102 is provided with a drive device that can read these media.

  The main part of the controller of the present invention is composed of computer hardware and software executed by the CPU 120. Generally, such software is stored and distributed in a storage medium such as a CD-ROM 118 or FD 116, read from the storage medium by the CD-ROM drive 108 or FD drive 106, and temporarily stored in the hard disk 124. Alternatively, when the device is connected to the network, it is temporarily copied from the server on the network to the hard disk 124. Then, the data is further read from the hard disk 124 to the RAM in the memory 122 and executed by the CPU 120. In the case of network connection, the program may be directly loaded into the RAM and executed without being stored in the hard disk 124.

  The computer hardware itself and its operating principle shown in FIGS. 1 and 2 are general. Therefore, the most essential part of the present invention is software stored in a storage medium such as the FD 116, the CD-ROM 118, and the hard disk 124.

  As a general tendency, various program modules are prepared as a part of a computer operating system, and an application program generally calls a module in a predetermined arrangement and advances the processing when necessary. In such a case, the software itself for realizing the controller does not include such a module, and the controller is realized only in cooperation with the operating system on the computer. However, as long as a general platform is used, it is not necessary to distribute software including such modules, and the software itself not including these modules and the recording medium storing the software (and the software distributes on the network). Data signal) can be considered to constitute the embodiment.

[General description of control method]
Hereinafter, the theoretical configuration of the configuration of the present invention will be described first.

(Configuration of controller)
(2. Premise: Policy gradient reinforcement learning)
In the following, the Markov decision process (MDP) will be considered, and the controlled object (controller environment) is in a state transition probability (execution (control) u _t when it is in the state x _t at time t). The same applies to the case where the probability is x _{t + 1} ) and the reward function r _{t + 1} = r (x _t , u _t ) (r _{t + 1} = r (x _{t + 1} , x _t , u _t )) Can be discussed). In addition, about this reward function, it shall be predetermined according to the control target of control object. The mapping from the state input xεX to the action output uεU is called a policy and is expressed stochastically as described below. The strategy is defined by the parameter θ.

Here, the conventional PGRL algorithm is reviewed and the main idea of the new algorithm of the present invention is presented. A discrete time MDP having a finite set of state X∋x and action U∋u is defined by the following state transition probability p and reward function r _{+ 1} .

Here, for simplicity of description, x _{+ 1} is the next state given by the action u in the state x, and r _{+ 1} is an immediate reward observed in x _{+ 1} [Non-Patent Document]. 10, Non-Patent Document 11]. x _{+ k} and u _{+ k} are respectively a state and an action which are k steps from the state x, and vice versa if the subscript is −k. The decision (in MDP) is made according to the following probabilistic policy π parameterized by θεR ^d .

  The following strategy π is assumed to be differentiable with respect to θ for all x∈X and u∈U.

Furthermore, the following assumptions are made.
(Assumption 1)
A Markov chain M (θ) having a state transition probability p and a stochastic policy π as follows is ergodic (irreducible and aperiodic) for all policy parameters.

  Thus, there is only one steady distribution:

  This steady distribution is independent of the initial state and satisfies the following equation.

  Here, the following equation holds.

The purpose of PRGL is to find a policy parameter θ ^* that maximizes the average of immediate rewards, called “average reward” below.

  Under Assumption 1, the average reward is shown to be independent of the initial state x and equal to the following equation [10]:

  The policy gradient RL algorithm updates the policy parameter θ in the direction of the average reward R (θ) gradient for θ shown in the following equation.

  Hereinafter, it is often referred to simply as policy gradient (PG). This policy gradient is given as follows.

  Deriving the partial derivative of the logarithm of the steady distribution shown in the following equation is not easy.

  Therefore, the conventional PG algorithm [Non-Patent Document 1, Non-Patent Document 2] uses another expression of PG.

  Here, the action value function Q and the state value function V are expressed as follows, respectively, with a discount rate γ∈ [0, 1) [Non-patent Document 11].

  The contribution of the second term of equation (4) becomes smaller as γ approaches 1 [Non-patent document 1], and the conventional algorithm [Non-patent document 1, Non-patent document 2] should be γ˜1. Thus, PG is approximated only from the first term. Such bias due to omission becomes smaller when the discount rate γ approaches 1, but the variance of estimation increases.

  Here, the present invention proposes another approach. There, the partial differential (LSDG) of the logarithm of the steady distribution of the following equation is estimated, and equation (3) is used to derive PG.

A striking feature is that no value function needs to be learned, and therefore the algorithm has nothing to do with the trade-off between bias and variance in the selection of discount rate γ.
(3. Estimation of partial derivative of logarithm of stationary distribution)
In the following, we propose an LSDG estimation algorithm, LSDG (λ), based on the least square method. For this purpose, we formulate the inverse process of the Ergotian Markov chain M (θ) and show that LSDG can be estimated in the framework of the TD method [Non-patent document 12, Non-patent document 13, Non-patent document 14]. .
(3.1 Properties of forward and backward Markov chains)
Using Bayesian theory, the backward probability from the current state to the past state and action pair is expressed by the following equation.

  The following posterior probability q depends on the prior distribution p.

  As will be described below, when the prior distribution p follows the steady distribution d and the policy π, the posterior distribution q is called a steady reverse probability, and a subscript B (θ) is added.

The Markov chain—both M (θ) and B (θ) —is closely related as described in the following two theorems.
(Theorem 1)

(Proof)
The following steady distribution is applied to both sides of Equation (5).

Then, summing over all possible actions u ₋₁ ∈U, we get:

  Then, when the sum is taken for the possible state xεX on both sides of the equation (7), the following equation is established.

  This establishes the following two points. (I) B (θ) has the same steady distribution as M (θ), and (ii) B (θ) has the same irreducible property as M (θ).

This suggests that (iii) B (θ) has the same aperiodic nature as M (θ). Equation (6) is directly proved by (i)-(iii) [Non-Patent Document 15].
(Theorem 2)

(Proof)
By substituting the properties of the Markov chain and equation (5), the following relationship is obtained.

  This proves that the equation (8) holds in the case of finite K. Since the following equation is derived from Theorem 1, it is immediately proved that the K → ∞ limit of equation (8) holds.

Theorem 1 and Theorem 2 indicate that samples from the forward Markov chain M (θ) can be used for estimation on the backward Markov chain B (θ) as they are under a state distribution that converges to a steady distribution. So it ’s important. These can be used later.
(3.2 TD (Temporal Difference) learning method for LSDG of Markov chain from reverse direction to forward direction)
LSDG is decomposed as follows using equation (5).

  Equation (10) implies that LSDG in state x is an accumulation of infinite intervals of backward Markov chain B (θ) from the logarithmic partial differential state x of the policy represented by the following equation: .

  From Equations (9) and (10), LSDG is estimated by TD learning [12] for the backward TD δ as follows for the backward Markov chain B (θ) rather than M (θ). Can be done.

  Here, the first two terms are the actual observation value one step before the logarithmic partial differentiation of the policy at B (θ) and the LSDG one step before, and the LSDG in the current state is the last term. .

  Using the eligibility decay rate λε [0,1] and the backward tracking time step KεN, equation (10) is generalized as follows:

  Even if the setting is not as described above, if a large λ or K is used, such minimization is more effective for the non-Markov effect as in the case of TD (λ) learning for the conventional value function. It is not sensitive [Non-Patent Document 16].

To fill the gap between theoretical assumptions and real-world applications, one of the following two strategies needs to be taken. (I) If λ˜1, K is not set to a very large integer, (ii)) If K˜t, λ is not set to 1, where t is the real forward Markov chain The current time step.
(3.3 LSDG Estimation Algorithm: Least Square Method of Restricted Reverse Direction TD (λ))

  This 3.3 proposes a [LS17], [13], [14], LSDG estimation algorithm, and LSDG (λ) based on the least square method. This simultaneously attempts to reduce the mean square and meet the constraints.

  Therefore, the function to be minimized is the following expression (13).

  Here, the second term on the right-hand side is for the constraint condition of Expression (12). Therefore, the partial differentiation of equation (13) is as follows.

  In a general RL problem, such a correlation exists, and thus an instrumental variable method is applied to eliminate such a bias [Non-patent Document 17, Non-Patent Document 13].

Hereinafter, to indicate the status of the time step t in the real Markov chain M (theta), to change the notation for x _t. The proposed LSDG estimation algorithm, LSDG (λ), makes the time step K going backwards the same as the time step t of the current state x _t under the qualifying decay rate λε [0,1). That is, the following holds.

  FIG. 3 is a diagram showing a procedure for obtaining LSDG (λ) as algorithm 1.

FIG. 4 is a flowchart showing the algorithm 1.
Referring to FIG. 4, first, in step 100, the following settings are made as a premise of processing.

  Subsequently, the following processing is performed as initialization processing (step S102).

  The time step t is set to t = 0 (step S104), and the following processing is repeated from t = 0 to t = T−1 (steps S106 to S116).

  First, if t = 0 in step S106, the initial state is observed (step S108), and then the following settings are made (S110).

  On the other hand, if t is not 0 in step S106, the following processing is performed (step S112).

  Following step S110 or S112, the following calculation is performed (step S114).

  In step S116, if t is smaller than T, the process returns to step S106. If t is equal to or greater than T, the process proceeds to step S118, and the following calculation is performed.

  Subsequently, an estimated value of LSDG is obtained by the following calculation.

(4. Policy update by LSDG estimation)
Here, a PGRL algorithm based on the LSDG estimation described above is defined.

  The policy parameters are updated by a probabilistic gradient method with an appropriate step size α.

Here, = indicates that substituting right side to the left side.
FIG. 5 is a diagram showing LSDG (λ) -PG as algorithm 2 as one of the simplest implementations of the PG algorithm using LSDG (λ). Here, the attenuation factor parameter β∈ [0, 1) is introduced in order to discard the past estimation given by the old value of θ.

  LSDG estimation with a function approximator constitutes another important content. That is, how to set the basis function φ (x) particularly in the continuous state problem.

Therefore, the following theorem is useful.
(Theorem 3)

(Proof)
This is proved by the following equation.

FIG. 6 is a flowchart corresponding to FIG.
Referring to FIG. 6, first, in step 200, the following setting is made as a premise of processing.

  Subsequently, the following processing is performed as initialization processing (step S202).

  The time step t is set to t = 0 (step S204), and the following processing is repeated from t = 0 to t = T-1 (steps S206 to S216).

  First, if t = 0 in step S206, the initial state is observed (step S208), and then the following settings are made (S210).

  On the other hand, if t is not 0 in step S206, the following processing is performed (step S212).

  Following step S210 or S212, the following calculation is performed (step S214).

  In step S116, if t is smaller than T, the process returns to step S206. If t is greater than or equal to T, the process proceeds to step S220, and the policy is updated by performing the following calculation. .

FIG. 7 is a conceptual diagram of the configuration of the controller of the present invention.
The controller of the present invention performs processing, that is, processing for giving a control signal to a control target, and observes the state quantity of the control target with an observer (for example, a position sensor, an angle sensor, an acceleration sensor, an angular acceleration sensor, etc.) Then, “logarithmic partial derivative of stationary distribution” (LSDG) is estimated from the observation result, the policy parameter is updated, and the policy is updated accordingly. The controlled object is further controlled by the updated policy.
(5. Results of numerical calculations)
It has a finite set of grids X = {1,..., M} and two possible actions U = {L, R} (one grid movement to the left (L) or right (R)) In the “one-dimensional torus-like grid space”, we verified the performance of our proposed algorithm. This is a typical m-state MDP task, and the state transition probabilities are given as follows.

Here, x = 0 and x = m (x = 1 and x = m + 1) are the same state, and p _i ∈ [0, 1] (i = 1,..., M) depends on the task. It is a constant. In our numerical calculations, the stochastic strategy is represented by the following sigmoidal function:

Here, the state feature vector phi (1), ..., all elements of the φ ^(m) ∈R m was taken out independently for each simulation from a steady normal distribution N (0,1 ^2). This was to verify how the parameterization of the stochastic policy affects the performance of our algorithm. The state feature vector φ (x) was also used as a basis function for LSDG estimation. Each simulation performed one episode consisting of more than 10 ⁵ time steps.

  First, we verified how accurately LSDG (λ) estimates the following “logarithmic partial derivative of stationary distribution” (LSDG) regardless of the problem setting and policy parameter θ.

To achieve this goal, task-dependent constants p ₁ ,..., P _m were chosen independently from the uniform distribution in the interval [0.7, 1] and were fixed in each simulation. The policy parameter θ was randomly selected according to the normal distribution N (0, 0.5 ² ) and fixed in each simulation.

  FIG. 8 shows a typical time course of the estimated LSDG when m = 3. Here, nine different colors indicate different elements of the LSDG. A solid line indicates a value estimated by LSDG (0), and a dotted line indicates an analytical solution of LSDG.

  As shown in FIG. 8, the estimation by LSDG (0) converges to an analytical solution in about 100 time steps when m = 3.

  A 7-state task was used to investigate the effect of the eligibility decay rate λ. In order to evaluate the average performance for different settings, the “relative error” criterion defined below was adopted.

Here, Ω ^* is the optimum parameter defined in Theorem 3, and was calculated analytically. FIGS. 9 and 10 show the average time course for 200 simulations of relative error for λ = 0, 0.3, 0.9 and 1. FIG. The only difference between these two figures is the number of elements of the feature vector φ (x).

The feature vector φ (x) εR ⁷ used in FIG. 9 is appropriate and sufficient to distinguish different states, while the feature vector φ (x) ε used in FIG. R ⁶ is not appropriate. These results are consistent with theoretical expectations. That is, if the basis function is appropriate (FIG. 9), λ can be set to an arbitrary value other than λ = 1, whereas λ is set to a larger value other than λ = 1 otherwise. There is a need (FIG. 10).

Finally, LSDG (λ) -PG was compared with other conventional PG methods in a three-state task. Here, the state transition probability is set to p _i = 1 for all iε {1, 2, 3}.

  FIG. 11 is a diagram showing setting of rewards in this task. Here, there are two types of rewards. The constant “r = (±) 2” and the variable “r = (±) s”. The variable s was randomly set from the uniform distribution in the interval [0.8, 1) in each simulation. Note that the reward s defines the minimum value of γ to find the optimal strategy as follows:

  Therefore, setting γ is important and difficult for this task. Two algorithms were adopted as the baseline of the performance of the conventional PG method: GPOMDP [Non-Patent Document 1] and Konda's actor-critic method [Non-Patent Document 5].

  FIG. 12 shows the average performance of the three methods for 100 simulations. Error bars indicate standard deviation for 100 times. Here, the performance was evaluated by 3R (θ) / (2-2s). That is, the average reward is normalized by the upper limit value (2-2s) / 3. The results show that the LSDG (λ) -PG method is superior to other methods.

  As described above, the present invention utilizes the fact that the actual forward and backward Markov chains are closely related and share a common property in the theorem. Using these, LSDG (λ) was proposed as an algorithm for estimating the logarithmic partial derivative (LSDG) of a stationary distribution, and LSDG (λ) -PG was proposed as a PG algorithm using LSDG estimation. Experimental results show that LSDG (λ) can operate with a qualified decay rate λε [0,1) and LSDG (λ) -PG can learn independently of discount rate γ. .

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a conceptual diagram which shows an example of the system 1000 using the controller with which the control method and control program of this invention are applied. FIG. 2 is a block diagram showing the configuration of a computer 100. FIG. 5 is a diagram showing a procedure for obtaining LSDG (λ) as algorithm 1; 3 is a flowchart showing Algorithm 1. It is a figure which shows LSDG ((lambda))-PG. 6 is a flowchart corresponding to FIG. It is a conceptual diagram of the structure of the controller of this invention. FIG. 6 is a diagram illustrating a typical time course of estimated LSDG when m = 3. It is a figure which shows the time passage of the relative error of LSDG estimated using sufficient feature vector when m = 7. It is a figure which shows the time passage of the relative error of LSDG estimated using the insufficient feature vector when m = 7. It is a figure which shows the setting of the reward in a task. FIG. 6 shows the average performance of the three methods for 100 simulations.

Explanation of symbols

  100 computer, 102 computer main body, 104 display, 106 FD drive, 108 CD-ROM drive, 110 keyboard, 112 mouse, 118 CD-ROM, 120 CPU, 122 memory, 124 hard disk, 128 communication interface, 200 controlled device, 1000 system.

Claims (9)

When the time evolution of the target system is described as a forward Markov decision process , a policy that is stochastically expressed as a control law for the state of the system is enhanced by the policy gradient method by observing the state quantity of the system A controller to learn,
Control signal generating means for generating a control signal for controlling the system based on the strategy;
State quantity detection means for observing the state quantity of the system;
Reward value acquisition means for acquiring a reward value depending on the state and the control signal in a predetermined relationship;
When the policy is defined by a policy parameter that is a parameter that defines the probabilistic policy, the logarithm of the steady distribution of the state distribution of the system is based on the state quantity and the control signal at each time step. by estimating the logarithmic normal distribution partial derivative is a partial differential of the policy parameters, a policy gradient estimation means for estimating a gradient of the measures,
Based on the estimation result obtained by the compensation value and the policy gradient estimation means, the logarithmic normal distribution partially differentiating the previous SL side measures parameters in the direction of the average earnings partial differential estimated using it to slightly changed, the measures A controller comprising policy updating means for updating. The policy gradient estimation means is a constraint condition derived by a condition that the sum of the distribution of the states with respect to the state is constant, and an expected value for a forward Markov chain of the logarithmic steady distribution partial differentiation is 0 The controller according to claim 1, wherein the logarithmic stationary distribution partial derivative is estimated by TD learning for a backward Markov chain under the constraint that   In the TD learning, i) the sum of the observed value one step before the logarithmic partial differentiation of the policy in the backward Markov chain, the logarithmic steady distribution partial differentiation one step before, and the logarithmic stationary of the current state When the difference from the distribution partial derivative is δ, the expected value of the square of δ for the forward Markov chain, and ii) the square of the expected value of the logarithmic steady distribution partial differentiation for the forward Markov chain The controller according to claim 2, wherein the logarithmic stationary distribution partial derivative is estimated by minimizing a sum of the two. When the time evolution of the target system is described as a forward Markov decision process , a policy that is stochastically expressed as a control law for the state of the system is enhanced by the policy gradient method by observing the state quantity of the system A control method to learn,
A control signal generating step for generating a control signal for controlling the system based on the strategy;
A state quantity detection step of observing the state quantity of the system;
A reward value acquiring step of acquiring a reward value depending on the state and the control signal in a predetermined relationship;
When the policy is defined by a policy parameter that is a parameter that defines the probabilistic policy, the logarithm of the steady distribution of the state distribution of the system is based on the state quantity and the control signal at each time step. by estimating the logarithmic normal distribution partial derivative is a partial differential of the policy parameters, a policy gradient estimation step of estimating the slope of the measures,
Based on the estimation result by the policy gradient estimating step and the compensation value, by updating the previous SL side measures parameters the direction of the gradient of the average earnings expressed on the basis of the logarithmic normal distribution partial derivative, updating the measures And a policy updating step. The policy gradient estimation step is a constraint condition derived by a condition that the sum of the distribution of the states with respect to the state is constant, and an expected value for a forward Markov chain of the logarithmic steady distribution partial differentiation is 0 The control method according to claim 4, further comprising the step of estimating the logarithmic steady distribution partial derivative by TD learning for a backward Markov chain under the constraint that In the TD learning, i) the sum of the observed value one step before the logarithmic partial differentiation of the policy in the backward Markov chain, the logarithmic steady distribution partial differentiation one step before, and the logarithmic stationary of the current state When the difference from the distribution partial derivative is δ, the expected value of the square of δ for the forward Markov chain, and ii) the square of the expected value of the logarithmic steady distribution partial differentiation for the forward Markov chain The control method according to claim 5, wherein the logarithmic steady distribution partial derivative is estimated by minimizing a sum of and. When the time evolution of the target system is described as a forward Markov decision process , a policy that is stochastically expressed as a control law for the state of the system is enhanced by the policy gradient method by observing the state quantity of the system A program for causing a computer to execute a learning control method,
A control signal generating step for generating a control signal for controlling the system based on the strategy;
A state quantity detection step of observing the state quantity of the system;
A reward value acquiring step of acquiring a reward value depending on the state and the control signal in a predetermined relationship;
When the policy is defined by a policy parameter that is a parameter that defines the probabilistic expression, the logarithm of the steady distribution of the state distribution of the system is based on the state quantity and the control signal at each time step. by estimating the logarithmic normal distribution partial derivative is a partial differential of the policy parameters, a policy gradient estimation step of estimating the slope of the measures,
Based on the estimation result by the policy gradient estimating step and the compensation value, by updating the previous SL side measures parameters the direction of the gradient of the average earnings expressed on the basis of the logarithmic normal distribution partial derivative, updating the measures A control program for causing a computer to execute a control method, comprising: The policy gradient estimation step is a constraint condition derived by a condition that the sum of the distribution of the states with respect to the state is constant, and an expected value for a forward Markov chain of the logarithmic steady distribution partial differentiation is 0 The control program according to claim 7, further comprising the step of estimating the logarithmic steady distribution partial derivative by TD learning for a backward Markov chain under the constraint that In the TD learning, i) the sum of the observed value one step before the logarithmic partial differentiation of the policy in the backward Markov chain, the logarithmic steady distribution partial differentiation one step before, and the logarithmic stationary of the current state When the difference from the distribution partial derivative is δ, the expected value of the square of δ for the forward Markov chain, and ii) the square of the expected value of the logarithmic steady distribution partial differentiation for the forward Markov chain The control program according to claim 8, wherein the logarithmic steady distribution partial derivative is estimated by minimizing a sum of the two.