Savo G. Glisic - Artificial Intelligence and Quantum Computing for Advanced Wireless Networks

For all filters in a given layer, we will assume that the order M lis the same. The activation value representing the output of a neuron in a layer, is given by the corresponding vector of delayed activations written as . Again, at the edges we have and . Instead of Table 3.1, a complete set of definitions is summarized in Table 3.2. The form of the two tables demonstrates a high level of similarity.

An interesting, more insightful, representation of the FIR network is derived by using a concept known as unfolding in time . The general strategy is to remove all time delays by expanding the network into a larger equivalent static structure.

Figure 3.5 Finite impulse response (FIR) neuron and neural network.

Table 3.2 Finite impulse response (FIR) multi‐layer network notation.

	Weight connecting neuron i in layer l − 1 to neuron j in layer l
	Bias weight for neuron j in layer l
	Summing junction or neuron j in layer l
	Activation value for neuron j in layer l
	Vector of delayed activation values
	i ‐th external input to network
	i ‐th output of network

Figure 3.6 Finite impulse response (FIR) network unfolding.

For the network shown in Figure 3.6, all connections are made by second‐order (three tap) FIRs. Although at first sight it looks as though we have only 10 connections in the network, in reality there are a total of 30 variable filter coefficients (not counting five bias weights). Starting at the output, each tap delay can be interpreted as a “virtual neuron,” whose input is delayed by the given number of time steps. A tap delay can be “removed” by replicating the previous layers of the network and delaying the input to the network as shown in Figure 3.6. The procedure is then carried on backward throughout each layer until all delays have been removed. The final unfolded structure is depicted in the bottom of Figure 3.6.

For supervised learning with input sequence x( k ), the difference between the desired output at time k and the actual output of the network is the error

(3.17)

The total squared error over the sequence is given by

(3.18)

The objective of training is to determine the set of FIR filter coefficients (weights) that minimizes the cost J subject to the constraint of the network topology. A gradient descent approach will be utilized again in which the weights are iteratively updated.

For instantaneous gradient descent , FIR filters may be updated at each time slot as

(3.19)

where is the instantaneous gradient estimate, and μ is the learning rate. However, deriving an expression for this parameter results in an overlapping of number of chain rules. A simple backpropagation ‐ like formulation does not exist anymore.

Temporal backpropagation is an alternative approach that can be used to avoid the above problem. To discuss it, let us consider two alternative forms of the true gradient of the cost function:

(3.20)

Note that

only their sum over all k is equal. Based on this new expansion, each term in the sum is used to form the following stochastic algorithm:

(3.21)