JPH0756877A - Neural network device - Google Patents

Abstract

PURPOSE:To enable effective learning by approximating the value of an reverse propagating signal required for learning to an ideal value by varying the output amplitudes of forward and backward signals. CONSTITUTION:The forward signal is inputted from a node 2A and outputted from a node 2B through an adjusting part 6A while multiplying weight at a multiplier 4A. At the adjusting part 6A, the size of the signal amplitude is changed corresponding to the advanced level of learning. The adjusting part 6A is operated so as to enlarge the amplitude when the learning is not advanced since the output of the multiplier 4A is fine, for example. Oppositely, when the learning is not advanced since the output of the multiplier part 4A is excess beyond a dynamic range, the adjusting part 6A is operated so as to reduce the amplitude. The reverse propagate signal is inputted from a node 2C, passed through an adjusting part 6B while multiplying weight at a multiplier 4B and outputted from a node 2D. Then, learning is enabled without controlling the signal amplitudes of forward and backward propagate signals by the dynamic range.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network device.

[0002]

2. Description of the Related Art In the field of recognition, control, etc., it is expected to introduce a neural network that performs parallel processing instead of conventional sequential processing. Has become. Currently, research on hardware implementation of neural networks is being actively conducted in Japan and overseas (Masahiko Koike: “Toward hardware implementation of neural networks” IEICE Research Report ICD92-12, Vol.92, No. .74 (1992)).

Neural networks are classified according to their structures and differences in learning rules. Generally, a multi-layered network is backpropagated (DLRumelhart,
JLMcClelland and PDP Research Group: "Parallel Di
stributed Processing Vol.1 "chapter 8, The MIT Pre
Many are taught by ss (1986)).

Various approaches have been taken to realize a neural network by hardware, but in order to downsize the device and parallelize the processing, signals are converted into current value, voltage value, light intensity. Analog system neural networks, which are dealt with by others, are considered promising.

However, these devices are limited in the dynamic range of the device. For example, when considering an analog multiplier made of electronic circuits, the linearity is deteriorated when the input exceeds a certain range, so that the operating range where a certain degree of linearity is guaranteed is limited (PRGray and RG Meyer: " Analy
sis and Design of Analog Integrated Circuits ", chap
ter 10, John Wiley & Sons, (1984)). The difference from the ideal calculation result due to the non-linearity of the multiplier is considered to be an error in the multiplication result. Multiplier error is a major cause of poor learning convergence in neural networks (T. Morishita, Y. Tamura and T. Otuki: "A BiCMOS.
Analog Neural Network with Dynamically Updated We
ights ", 1990 IEEE International Solid-State Circuit
s Conference Digest of Technical Papers, Vol.33, p
p.142-143, (1990)). Therefore, in order to avoid this, it is necessary to match the signal level to the dynamic range of the device. However, this causes a new problem that the absolute value of the signal deviates from the ideal calculation result due to the relative difference between the absolute value of the signal and the magnitude of other variables. This problem is significant in the back-propagating signal needed for learning.

As described above, in the conventional analog neural network apparatus, since the dynamic range of the input and output of the apparatus is fixed, the learning may not proceed depending on the magnitude of the input and output signals. This has been an obstacle to the realization of an analog neural network device with sufficient learning performance.

[0007]

As described above, in the conventional analog neural network device, the input of the device,
Since the output dynamic range is limited and the back-propagation signal necessary for learning deviates from the ideal value, it was very difficult to realize sufficient learning performance.

The present invention provides a neural network device that enables effective learning by making the output amplitudes of the forward signal and the backward signal variable so that the value of the back-propagation signal required for learning approaches an ideal value. Aim to achieve.

[0009]

In order to achieve the above object, a neural network device of the invention included in the present application comprises first and second neuron layers having a plurality of neurons, and first and second neuron layers thereof. And a synapse layer composed of synapses connecting neurons in the two neuron layers, and using the connection weight of each synapse that is sequentially updated based on the error signal calculated based on the teacher signal given from the outside, Performing an operation on the output signal of the first neuron layer,
In a neural network device that transfers the operation result signal to the second neuron layer, if learning does not proceed, the operation result signal transferred to the second neuron layer is based on a control signal given from the outside. It is characterized by having a changing means for changing.

Further, it has a forward signal path from the input layer to the output layer and a reverse signal path for transmitting an error signal calculated based on a teacher signal given from the outside from the output layer to the input layer side. In the multi-layered neural network that has the function of changing the synaptic connection weight value by obtaining the update amount of the synaptic connection weight between the output of the forward signal of the neuron and the output of the backward signal of the subsequent neuron, the output of the previous neuron A variable means for varying the output value of the output of the forward signal as a result of multiplying by the weight and the output value of the output of the backward signal as a result of multiplying the error signal from the neuron in the subsequent stage by the weight by a control signal given from the outside Is characterized by.

Further preferably, the variable means outputs a forward signal output as a result of multiplying an output of the preceding neuron by a weight and an output of a reverse signal as a result of multiplying an error signal from the subsequent neuron by a weight. It is characterized in that it is configured by means for making the value independently variable for each layer by a control signal given from the outside.

Further preferably, the variable means outputs the output of the forward signal as a result of multiplying the output of the neuron in the preceding stage by a weight and the output of the output of the backward signal as a result of multiplying the error signal from the neuron in the subsequent stage by the weight. It is characterized in that it is configured by means for independently changing the forward signal and the back propagation signal by a control signal given from the outside.

[0013]

According to the neural network device of the present invention, since it has a function of adjusting the output amplitudes of the forward signal and the backward signal according to the progress of learning and the like,
The value of the back-propagation signal required for learning can be brought close to the ideal value within the operating range of the device, which is effective in realizing a neural network device with sufficient learning performance.

[0014]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing the basic configuration of the synapse unit of the neural network system of the present invention. Here, the synapse part means a part 8 that connects one neuron to the other neuron in the network of FIG.

The synapse section 1 has four signal nodes 2A and 2A.
B, 2C, 2D, a storage unit 3 for synapse connection values, a multiplication unit 4A for forward signals and synapse connection values, a multiplication unit 4B for backpropagation signals and synapse connection values, and a learning control unit 5,
It is composed of output signal adjusting units 6A and 6B.

The learning proceeds so that the actual output signal approaches the teacher signal, and the learning control section uses the input from the node 2A and the back-propagation signal (error signal) from the node 2C. The connection weight is calculated, and the value of the connection weight (connection value) at the synapse is updated and stored in the connection value storage unit 3.

Here, the forward signal is input from the node 2A, weighted by the multiplier 4A, passed through the adjusting unit 6A, and output from the node 2B. The adjusting unit 6A changes the magnitude of the signal amplitude according to the progress of learning. For example, when the output of the multiplication unit 4A is very small and learning does not proceed because the dynamic range is not effectively utilized, the adjustment unit 6A operates to increase the amplitude. Conversely, when the learning does not proceed because the output of the multiplication unit 4A is too large and exceeds the dynamic range, the adjustment unit 6A operates to reduce the amplitude.

The backpropagation signal is input from the node 2C, weighted by the multiplier 4B, passed through the adjusting unit 6B, and output from the node 2D. Also in this case, the same operation as that of the adjusting unit in the output of the forward signal is performed, and the adjusting unit 6B performs a calculation of adjusting the amplitude of the output of the multiplier according to the control signal from the outside.

With this configuration, it is possible to effectively perform learning without limiting the signal amplitudes of the forward propagation signal and the backward propagation signal to the dynamic range of the device.
In the learning method that reduces the error represented by the back propagation method by the steepest descent method, the weight of the synapse coupling is updated based on the error signal that propagates back from the output layer side to the input layer side.

In the three-layered network shown in FIG. 2, an error signal E that propagates backward to learn the synapse Wij that connects the input layer neuron i and the intermediate layer neuron j.
j is expressed as Ej = Fj '* [Sigma] k Fk' * (Tk-Ok) * Wjk (1). Here, each symbol is: Fj ': differential value of the intermediate layer neuron j Fk': differential value of the output layer neuron k Tk: teacher signal to the output layer neuron k Ok: output value of the output layer neuron k Wjk: intermediate layer A synaptic connection value between the neuron j and the output layer neuron k is represented.

In actual hardware, the following non-ideal calculation is performed due to the limitation of the dynamic range and the like. f ′ = dF ′ (2) Σ ′ = σΣ (3) (X−Y) ′ = m (X−Y) (4) In these equations, the left side is a value calculated by an actual device. Yes, the right side is a theoretically obtained value. Further, (2) represents a differential operation, (3) represents an addition operation, and (4) represents a subtraction operation. Using these, equation (1) can be rewritten as follows.

Ej = fj '× Σ'kfk' × (Tk-Ok) '× Wjk = dj Fj' × σΣk DK Fk '× m (Tk-Ok) × Wjk = dj σdk mFj' × Σk Fk '× ( Tk-Ok) × Wjk (5) In this, d and m are fixed at the time of hardware design,
σ can be easily adjusted by adjusting the bias value, the gain, and the like.

In particular, the equation (3) can be transformed into Σ ′ = σΣ = Σσ ′ (6) (σ ′: error signals of each of a plurality of signals input to the device).

Therefore, by changing the value of σ ',
The coefficient of the equation (5) can be adjusted. For example, effective learning can be performed by setting the coefficient to 1 so as to approximate the ideal calculation, or by increasing the coefficient so that the back propagation signal is not buried in noise. Furthermore, by changing the value of σ, the learning coefficient η used in the backpropagation method can be introduced in an arbitrary size without multiplication operation, and the value of η together with the value of σ can be different for each layer. Can be changed. With respect to the forward signal, it also has a function of changing the slope of the nonlinear function of the next-stage neuron by changing the value of σ. Also,
Since changing the value of σ has the effect of matching the signal to the dynamic range of the device, it also has the effect that the network scale can be changed without drastically changing the device.

In the above embodiment, the configuration example in which the adjusting unit in the synapse is controlled according to the progress of learning has been described, but the following modified examples are also possible. FIG. 3 shows a second configuration example of the present invention. This is, in addition to the basic configuration of the neural network device described in the above-described embodiment, an adjustment unit 6C that adjusts the amplitude of the forward signal input to the learning control unit 5 or the multiplication unit 4A, the learning control unit 5 or the multiplication unit 4.
An adjustment unit 6D that adjusts the amplitude of the back-propagation signal input to B,
A control unit 6E for controlling the amplitude of the output signal of the combined value storage unit,
It is characterized by having 6F. As a result, it is possible to control the signal amplitude of the combined value input to the multiplication unit, and to ensure an operation suitable for the operation range of the device.

For example, by comparing the dynamic range of a signal given at the time of hardware design with the propagating signal amplitude, the adjusting unit may be controlled when the signal amplitude exceeds the dynamic range or is buried in the noise level. It is possible.

Further, the learning control unit 5 calculates the coupling value at the synapse. If this coupling value is close to the maximum value, the coupling value will be saturated even if the learning progresses, and the learning apparently proceeds. It may disappear. In such a case, by controlling the adjusting unit to amplify the signal of the combined value, the apparent combined value can be increased and the learning can be further advanced.

Depending on the neural network device, it is possible to provide the synapse coupling value discretely. However, at this time, the coupling value changes discretely, so that the intermediate value of the discrete points cannot be taken. , Learning progress may become unstable. Even in such a case, it is possible to finely adjust the forward output and the backward propagation output of the synapse by slightly changing each adjusting unit, and the resolution of the combined value can be apparently improved.

[0029]

As described above, according to the present invention, it is effective to realize the neural network device having sufficient learning performance by adapting to the limitation of the dynamic range of the device.

[Brief description of drawings]

FIG. 1 is a diagram showing a basic configuration example of the present invention.

FIG. 2 is a diagram for explaining a learning rule.

FIG. 3 is a diagram showing another configuration example of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Synapse part 2 ... Signal node 3 ... Connection weight storage part 4 ... Multiplication part 5 ... Learning control part 6 ... Output value adjustment part 7 ... Neuron 8 ... Synapse

Claims (4)

[Claims] 1. A teacher externally provided, comprising first and second neuron layers each having a plurality of neurons, and a synapse layer composed of synapses connecting neurons in the first and second neuron layers. The output signal of the first neuron layer is operated using the connection weight of each synapse that is sequentially updated based on the error signal calculated based on the signal, and the operation result signal is applied to the second neuron layer. The neural network device for transferring has a changing means for changing the operation result signal transferred to the second neuron layer based on a control signal given from the outside when the learning is stopped. Neural network device. 2. A forward signal path from the input layer to the output layer, and a backward signal path for transmitting an error signal calculated based on a teacher signal given from the outside to the input layer side, In the multilayer neural network having the function of changing the synapse connection weight value by obtaining the update amount of the synapse connection weight by the output of the forward signal of the neuron in the front stage and the output of the backward signal of the neuron in the rear stage, It has a variable means for changing the output value of the output of the forward signal as a result of multiplying the output by the weight and the output of the backward signal as a result of multiplying the error signal from the neuron in the subsequent stage by the weight by the control signal given from the outside. A neural network device characterized by the above. 3. The variable means sets the output value of the output of the forward signal as a result of multiplying the output of the neuron in the preceding stage by weight and the output value of the output of the backward signal as a result of multiplying the error signal from the neuron in the subsequent stage by the weight. 3. The neural network device according to claim 2, wherein the neural network device is constituted by means for independently changing each layer by a control signal given from the outside. 4. The variable means sets an output value of an output of a forward signal as a result of multiplying an output of a neuron in a preceding stage by a weight and an output value of an output of a backward signal as a result of multiplying an error signal from a neuron in a subsequent stage by weight. 3. The neural network device according to claim 2, wherein the forward signal and the back propagation signal are independently variable by a control signal given from the outside.