Savo G. Glisic - Artificial Intelligence and Quantum Computing for Advanced Wireless Networks

(3.7)

so that Δw = 2 μe x. From this, we have Δ w i= 2 μex i, which is the least mean square (LMS) algorithm.

In a multi‐layer network , we just formally extend this procedure. For this we use the chain rule

(3.8)

with leading to the weight update .

Parameters δ are derived recursively starting from the output layer:

(3.9)

where f ′ is the derivative of the sigmoid function of s . We have also used for the output layer . With this, at the output layer, each neuron has an explicit desired response, so we can write

(3.10)

Substituting into Eq. (3.9)yields .

To calculate the δ ′ s , we note that e Te is influenced through indirectly through all node values in the next layer. Referring to the upper part of Figure 3.3, we again employ the chain rule

(3.11)

with

(3.12)

Recalling that , we get In summary, we have

(3.13)

(3.14)

For the bias weight we note that in Eq. (3.13). The above processing is illustrated in Figure 3.4, indicating the symmetry between the forward propagation of neuron activation values and the backward propagation of δ terms.

Figure 3.4 Illustration of backpropagation.

Most often in engineering, prior to becoming a member of the observation set, the input signals to the neural network have gone through some form of filtering. This also coincides with the form of potential maintained at the axon hillock region of the neural cell. With this in mind, we may modify Eq. (3.1)as

(3.15)

By adding filtering operations, we have included the equally important temporal dimension in the static model. For our purposes, we will now be interested in adapting the filters. To this end, we assume a discrete FIR representation for each filter. This yields

(3.16)

with k being the discrete time index for some sampling rate Δ t , and w i( n ) being the coefficients for the FIR filters. In the following, we will represent the vector w i= [ w i(0), w i(1), … , w i( M )] and the delayed states as x i( k ) = [ x i( k ), x i( k − 1), … , x i( k − M )]. Now, a filter operation is written as the vector dot product w ix i( k ), with time implicitly included in the notation.

The top part of Figure 3.5shows a standard representation of an FIR filter as a tap delay line. Although this filter represents several biological processes, as well as many engineering solutions, for ease of reference to a real neuron network we will refer to an FIR filter as a synaptic filter or simply a synapse . The output of the neuron will be as before y ( k ) = f ( s ( k )) with f ( x ) = tanh ( x ), and we have added only a time index k .

We use the same approach to network modeling as in the previous section. Each link in the network is now created using an FIR filter (see Figure 3.5). The neural network no longer performs a simple static mapping from input to output; internal memory has now been added to simple static mapping from input to output. At the same time, since there are no feedback loops, the overall network is still FIR [2–5]. The notation now becomes .