Artificial Intelligence A Technique Which Enables Machines - IT Computer Science Assignment Help

Assignment Task:

Task:

Basics of Deep Learning:

Artificial Intelligence: A technique which enables machines to mimic human behaviour
Machine Learning is a subset of AI techniques which use statistical methods to enable machines to improve with experience
Deep Learning: Subset of ML which make the computation on multi-layer neural network feasible.

Artificial neural network (ANN) are modeled the same as the human brain. The human brain has a mind to think and analyze any task in a particular situation. But how can a machine think like that? For the purpose, an artificial brain was designed is known as a neural network. The neural network is made up many perceptrons.
The artificial neuron has input and output.

Representation of perceptron model mathematically

A neural network can be made with multiple perceptron’s. Where there are three layers-
Input layer: Input layers are the real value from the data.
Hidden layer: Hidden layers are between input and output layers where three or more layers are deep network.
Output layer: It is the final estimate of the output
Advantages of ANN
After the training of ANN, the data may give the result even with incomplete information.
If one or more cell is corrupt of ANN, it does not prevent ANN to generate output.
ANN has distributed memory that helps to generate the desired output and it can make a machine learnable.
ANN has a parallel processing capability, which means it can perform more than one task at the same time.
Disadvantages of ANN
It requires a processor with parallel processing power according to their structure.
Unexplained behaviour of the network is the main problem of ANN. ANN doesn't give a clue when it produces a probing solution.
For the determination of the structure of ANN, no specific rules are providing.
It's too typical to show the problem to the network
Deep Learning:
Deep learning is a collection of statistical techniques of machine learning for learning feature hierarchies that are actually based on artificial neural networks. Basically, it is a machine learning class that makes use of numerous nonlinear processing units so as to perform feature extraction as well as transformation. The output from each preceding layer is taken as input by each one of the successive layers.
So basically, deep learning is implemented by the help of deep networks, which are nothing but neural networks with multiple hidden layers.

• Linear Regression is a machine learning algorithm based on supervised learning. It performs a regression task. Regression models a target prediction value based on independent variables. It is mostly used for finding out the relationship between variables and forecasting
• Logistic regression is a classification algorithm used to assign observations to a discrete set of classes.Unlike linear regression which outputs continuous number values, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.
• The cost function is calculated as an average of loss function. It quantifies the error between predicted values and expected values and presents it in the form of a single real number
• Vectorization is the process of converting an algorithm from operating on a single value at a time to operating on a set of values at one time. by using a vectorized implementation in an optimization algorithm we can make the process of computation much faster compared to Unvectorized Implementation
• Forward propagation is how neural networks make predictions. Input data is “forward propagated” through the network layer by layer(Left to Right) to the final layer which outputs a prediction.
• As the name suggests, the input data is fed in the forward direction through the network. Each hidden layer accepts the input data, processes it as per the activation function and passes to the successive layer.
• In order to generate some output, the input data should be fed in the forward direction only. The data should not flow in reverse direction during output generation otherwise it would form a cycle and the output could never be generated. Such network configurations are known as feed-forward network. The feed-forward network helps in forward propagation.
• At each neuron in a hidden or output layer, the processing happens in two steps:
• Pre-activation: it is a weighted sum of inputs i.e., the linear transformation of weights w.r.t to inputs available. Based on this aggregated sum and activation function the neuron makes a decision whether to pass this information further or not.
• Activation: the calculated weighted sum of inputs is passed to the activation function. An activation function is a mathematical function which adds non-linearity to the network. There are four commonly used and popular activation functions — sigmoid, hyperbolic tangent(tanh), ReLU and Softmax.
• Backward Propagation is the practice of fine-tuning the weights(Right to Left) of a neural net based on the error rate (i.e. loss) obtained in the previous iteration.
• Back-propagation is the essence of neural net training. It is the practice of fine-tuning the weights of a neural net based on the error rate (i.e., loss) obtained in the previous epoch (i.e., iteration). Proper tuning of the weights ensures lower error rates, making the model reliable by increasing its generalization.
• We have a model that does not give accurate predictions and that is attributed to the fact that its weights have not been tuned yet. We also have the loss. Back-propagation is all about feeding this loss backwards in such a way that we can fine-tune the weights based on which. The optimization function such as gradient descent will help us find the weights that will — hopefully — yield a smaller loss in the next iteration.
• The overall steps are:
• In the forward propagate stage, the data flows through the network to get the outputs
• The loss function is used to calculate the total error
• Then, we use backward propagation algorithm to calculate the gradient of the loss function with respect to each weight and bias
• Finally, we use gradient descent to update the weights and biases at each layer
• We repeat above steps to minimize the total error of the neural network.
• Using the input variables x and y, the forward pass or propagation calculates output z as a function of x and y i.e. f(x,y).
• During backward pass or propagation, on receiving dL/dz (the derivative of the total loss, L with respect to the output, z), we can calculate the individual gradients of x and y on the loss function by applying the chain rule, as shown in the figure.
• An activation function defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the network. The role of the Activation Function is to derive output from a set of input values fed to a node (or a layer). The purpose of an activation function is to add non-linearity to the neural network
• Binary Step Function:
• Binary step function depends on a threshold value that decides whether a neuron should be activated or not. 
• The input fed to the activation function is compared to a certain threshold; if the input is greater than it, then the neuron is activated, else it is deactivated, meaning that its output is not passed on to the next hidden layer.
•  
• Binary Step Function
• Mathematically it can be represented as:
•  
• Here are some of the limitations of binary step function:
• It cannot provide multi-value outputs—for example, it cannot be used for multi-class classification problems. 
• The gradient of the step function is zero, which causes a hindrance in the backpropagation process.
• Linear Activation Function
• The linear activation function is also known as Identity Function where the activation is proportional to the input.
•  
• Linear Activation Function
• Mathematically it can be represented as:
•  
• However, a linear activation function has two major problems :
• It’s not possible to use backpropagation as the derivative of the function is a constant and has no relation to the input x. 
• All layers of the neural network will collapse into one if a linear activation function is used. No matter the number of layers in the neural network, the last layer will still be a linear function of the first layer. So, essentially, a linear activation function turns the neural network into just one layer.
• Non-linear activation functions solve the following limitations of linear activation functions:
• They allow backpropagation because now the derivative function would be related to the input, and it’s possible to go back and understand which weights in the input neurons can provide a better prediction.
• They allow the stacking of multiple layers of neurons as the output would now be a non-linear combination of input passed through multiple layers. Any output can be represented as a functional computation in a neural network.
• Now, let’s have a look at different non-linear neural networks activation functions and their characteristics.
• Sigmoid / Logistic Activation Function 
• This function takes any real value as input and outputs values in the range of 0 to 1.
• The larger the input (more positive), the closer the output value will be to 1.0, whereas the smaller the input (more negative), the closer the output will be to 0.0, as shown below
•  
• Sigmoid/Logistic Activation Function
• Mathematically it can be represented as:
•  
• Here’s why sigmoid/logistic activation function is one of the most widely used functions:
• It is commonly used for models where we have to predict the probability as an output. Since probability of anything exists only between the range of 0 and 1, sigmoid is the right choice because of its range.
• The function is differentiable and provides a smooth gradient, i.e., preventing jumps in output values. This is represented by an S-shape of the sigmoid activation function. 
• The limitations of sigmoid function are
• The function will have very small gradients. As the gradient value approaches zero, the network ceases to learn and suffers from the Vanishing gradient problem.
• The output of the logistic function is not symmetric around zero. So the output of all the neurons will be of the same sign. This makes the training of the neural network more difficult and unstable.
• ReLU Function
• ReLU stands for Rectified Linear Unit. Although it gives an impression of a linear function, ReLU has a derivative function and allows for backpropagation while simultaneously making it computationally efficient. The main catch here is that the ReLU function does not activate all the neurons at the same time. The neurons will only be deactivated if the output of the linear transformation is less than 0.
•  
• ReLU Activation Function
• Mathematically it can be represented as:
•  
• The advantages of using ReLU as an activation function are as follows:
• Since only a certain number of neurons are activated, the ReLU function is far more computationally efficient when compared to the sigmoid and tanh functions.
• ReLU accelerates the convergence of gradient descent towards the global minimum of the loss function due to its linear, non-saturating property.
• The limitations faced by this function are:
• The negative side of the graph makes the gradient value zero. Due to this reason, during the backpropagation process, the weights and biases for some neurons are not updated. This can create dead neurons which never get activated. 
• All the negative input values become zero immediately, which decreases the model’s ability to fit or train from the data properly.
• Softmax Function
• Softmax Function calculates the probabilities distribution of the event over ‘n’ different events, this function will calculate the probabilities of each target class over all possible target classes. The Softmax function is described as a combination of multiple sigmoids. It calculates the relative probabilities. Similar to the sigmoid/logistic activation function, the SoftMax function returns the probability of each class. 
• It is most commonly used as an activation function for the last layer of the neural network in the case of multi-class classification. 
•  
• Probability
• Mathematically it can be represented as:
•  
• Softmax Function
• And here are a few other guidelines to help you out.
• ReLU activation function should only be used in the hidden layers.
• Sigmoid/Logistic should not be used in hidden layers as it make the model more susceptible to problems during training.
• a few rules for choosing the activation function for your output layer based on the type of prediction problem that you are solving:
• Regression - Linear Activation Function
• Binary Classification - Sigmoid/Logistic Activation Function
• Multiclass Classification - SoftmaxMultilabel Classification - Sigmoid
• The activation function used in hidden layers is typically chosen based on the type of neural network architecture.
• Convolutional Neural Network (CNN): ReLU activation function.
• Recurrent Neural Network: Tanh and/or Sigmoid activation function.
• Bias is the simplifying assumptions made by the model to make the target function easier to approximate. 
•  
• with bias, the input to the activation function is 'x' times the connection weight 'w0' plus the bias times the connection weight for the bias 'w1'. This has the effect of shifting the activation function by a constant amount (b * w1).
• Variance is the amount that the estimate of the target function will change given different training data. Trade-off is tension between the error introduced by the bias and the variance
• Regularization is the process of regularizing the parameters that constrain, regularizes, or shrinks the coefficient estimates towards zero and hence reduce cost term. This technique discourages learning a more complex or flexible model, avoiding the risk of Overfitting.
• Regularization, significantly reduces the variance of the model, without substantial increase in its bias. So the tuning parameter ?, controls the impact on bias and variance. As the value of ? rises, it reduces the value of coefficients and thus reducing the variance. Till a point, this increase in ? is beneficial as it is only reducing the variance(hence avoiding overfitting), without loosing any important properties in the data.
• It is a useful technique that can help in improving the accuracy of your regression models
• There are various regularization techniques, some well-known techniques are L1, L2 and dropout regularization.
• L1 regularization: It adds an L1 penalty that is equal to the absolute value of the magnitude of coefficient, or simply restricting the size of coefficients
• L2 Regularization: It adds an L2 penalty which is equal to the square of the magnitude of coefficients
• DropOut is a Simple Way to Prevent Neural Networks from Overfitting. Dropout is a technique where randomly selected neurons are ignored during training. They are “dropped-out” randomly. This means that their contribution to the activation of downstream neurons is temporally removed on the forward pass and any weight updates are not applied to the neuron on the backward pass. Dropout may be implemented on any or all hidden layers in the network as well as the visible or input layer. It is not used on the output layer.
• The Exponentially Weighted Moving Average (EWMA) is a quantitative or statistical measure used to model or describe a time series. The moving average is designed as such that older observations are given lower weights. The weights fall exponentially as the data point gets older – hence the name exponentially weighted.
• The only decision a user of the EWMA must make is the parameter alpha. The parameter decides how important the current observation is in the calculation of the EWMA. The higher the value of alpha, the more closely the EWMA tracks the original time series.
• EWMA Formula
•  
• Gradient Descent is an optimization algorithm for finding a local minimum of a differentiable function. Gradient descent is simply used in machine learning to find the values of a function's parameters (coefficients) that minimize a cost function as far as possible.
• There are two types of Gradient Descent:
• Adam is an optimization algorithm that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data.
• Below are the benefits of using Adam
• Straightforward to implement, computationally efficient and Little memory requirements. Invariant to diagonal rescale of the gradients.
• Well suited for problems that are large in terms of data and/or parameters.
• Appropriate for non-stationary objectives and for problems with very noisy/or sparse gradients.
• Hyper-parameters have intuitive interpretation and typically require little tuning
• RMSprop stands for root mean square prop, which can also accelerate gradient descent. RMSprop uses the same concept of the exponentially weighted average of gradient as gradient descent with momentum but the difference is parameter update.
• Normalization:
• Normalization is an approach which is applied during the preparation of data in order to change the values of numeric columns in a dataset to use a common scale when the features in the data have different ranges. There are four normalization menthods which can be used in deep learning models.
• Batch Normalization
• Weight Normalization
• Layer Normalization
• Group Normalization
• Batch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.  It is a process to make neural networks faster and more stable through adding extra layers in a deep neural network. The new layer performs the standardizing and normalizing operations on the input of a layer coming from a previous layer. A typical neural network is trained using a collected set of input data called batch. Similarly, the normalizing process in batch normalization takes place in batches, not as a single input
• This approach is mainly related to internal covariate shift (ICS) where internal covariate shift means the change in the distribution of layer inputs caused when the preceding layers are updated. In order to improve the training in a model, it is important to reduce the internal co-variant shift. The batch normalization works here to reduce the internal covariate shift by adding network layers which control the means and variances of the layer inputs
• Advantages of Batch Normalization
• By Normalizing the hidden layer activation the Batch normalization speeds up the training process.
• It solves the problem of internal covariate shift. Through this, we ensure that the input for every layer is distributed around the same mean and standard deviation
• Disadvantages of Batch Normalization
• Calculation of these batch-level statistics is expensive and may subject to memory overhead, and significantly increases the time required to evaluate the gradient in some networks
• Because of Its dependence on batch size, it's really hard to replicate the model performance on a setting with very limited hardware resources.
• Weight normalization is a process of reparameterization of the weight vectors in a deep neural network which works by decoupling the length of those weight vectors from their direction. In simple terms, we can define weight normalization as a method for improving the optimisability of the weights of a neural network model.
• Advantages
• The advantages of weight normalization are mentioned below
• Weight normalization improves the conditioning of the optimisation problem as well as speed up the convergence of stochastic gradient descent.
• It can be applied successfully to recurrent models such as LSTMs as well as in deep reinforcement learning or generative models
• Disadvantages
• Compared with the others, might be unstable on training
• High dependence to input data
• Layer normalization is a method to improve the training speed for various neural network models. Unlike batch normalization, this method directly estimates the normalisation statistics from the summed inputs to the neurons within a hidden layer. Layer normalization is basically designed to overcome the drawbacks of batch normalization such as dependent on mini batches.
• Advantages
• The advantages of layer normalization are mentioned below:
• Layer normalization can be easily applied to recurrent neural networks by computing the normalization statistics separately at each time step
• This approach is effective at stabilising the hidden state dynamics in recurrent networks 
• Disadvantages
• Might be not good for CNN (Batch Norm is better in some cases)
• Group normalization can be said as an alternative to batch normalization. This approach works by dividing the channels into groups and computes within each group the mean and variance for normalization i.e. normalising the features within each group. Unlike batch normalization, group normalization is independent of batch sizes, and also its accuracy is stable in a wide range of batch sizes. 
• Advantages
• It has the ability to replace batch normalization in a number of deep learning tasks
• It can be easily implemented in modern libraries with just a few lines of codes
• Disadvantages
• when the batch size is significantly large, Group normalization does not scale and might not be able to match the performance of Batch Normalization.
• Orthogonalization is a system design property that ensures that modification of an instruction or an algorithm component does not create or propagate side effects to other system components. Orthogonalization makes it easier to independently verify the algorithms, thus reducing the time required for testing and development.
• Types of Neural Network: In this document we have covered two types of Neural Network they are
• Convolutional Neural Network
• Recurrent Neural Network
• A convolutional neural network (CNN) is a type of artificial neural network used in image recognition and processing that is specifically designed to process pixel data.
• CNNs are powerful image processing, artificial intelligence (AI) that use deep learning to perform both generative and descriptive tasks, often using machine vison that includes image and video recognition, along with recommender systems and natural language processing (NLP).
• Architecture of CNN:
•  
• The convolutional layer is the key component of convolutional neural networks, and is always at least their first layer. Its purpose is to detect the presence of a set of features in the images received as input. This is done by convolution filtering: the principle is to “drag” a window representing the feature on the image, and to calculate the convolution product between the feature and each portion of the scanned image.
• A pooling layer is another building block of a CNN. Its function is to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network. Pooling layer operates on each feature map independently. The most common approach used in pooling is max pooling.
• Max pooling is a pooling operation that selects the maximum element from the region of the feature map covered by the filter. Thus, the output after max-pooling layer would be a feature map containing the most prominent features of the previous feature map.  
•  
• The fully-connected layer is always the last layer of a neural network, convolutional or not — so it is not characteristic of a CNN.
• This type of layer receives an input vector and produces a new output vector. To do this, it applies a linear combination and then possibly an activation function to the input values received.
• Edge detection is an image processing technique for finding the boundaries of objects within images It works by detecting discontinuities in brightness These points where the image brightness varies sharply are called the edges (or boundaries) of the image.
• Padding is the amount of pixels added to an image when it is being processed by the kernel of a CNN. It defines how the border of a sample is handled. A (half) padded convolution will keep the spatial output dimensions equal to the input, whereas unpadded convolutions will crop away some of the borders if the kernel is larger than 1.
• Kernel Size:  Convolutional neural networks (CNN), are basically a stack of layers which are defined by the action of a number of filters on the input. Those filters are usually called kernels. The kernel size here refers to the widthxheight of the filter mask. The kernel is a matrix that moves over the input data, performs the dot product with the sub-region of input data, and gets the output as the matrix of dot products. Kernel moves on the input data by the stride value. If the stride value is 2, then kernel moves by 2 columns of pixels in the input matrix. In short, the kernel is used to extract high-level features like edges from the image.
• the concept of edge detection, taking an example of a simplified image.
•  
• A 6?6 image convolved with 3?3 kernel
• So if a 6*6 matrix convolved with a 3*3 matrix output is a 4*4 matrix. To generalize this if a ???? ? ???? image convolved with ???? ? ???? kernel, the output image is of size (???? ? ???? + 1) ? (???? ? ???? + 1).
• Consider the two input image arrangements as shown in the example below. For the first image, the center value is 3*5 + 2*-1 + 2*-1 + 2*-1 + 2*-1 = 7. The value 3 got increased to 7. For the second image, the output is 1*5+ 2*-1 + 2*-1 + 2*-1 + 2*-1 = -3. The value 1 got decreased to -3. Clearly, the contrast between 3 and 1 is increased to 7 and -3, which will in turn sharpen the image.
•  
• Channels: In a grayscale image, the data is a matrix of dimensions w×h, where w is the width of the image and h is its height. In a colour image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions w×h×c, where cc is the number of channels, that is, 3.
• A convolution layer receives the image (w×h×c) as input, and generates as output an activation map of dimensions w?×h?×c?. The number of input channels in the convolution is cc, while the number of output channels is c?. The filter for such a convolution is a tensor of dimensions  f×f×c×c?, where f is the filter size (normally 3 or 5).
• This way, the number of channels is the depth of the matrices involved in the convolutions. Also, a convolution operation defines the variation in such depth by specifying input and output channels.
• Inception Modules are used in Convolutional Neural Networks to allow for more efficient computation and deeper Networks through a dimensionality reduction with stacked 1×1 convolutions. The modules were designed to solve the problem of computational expense, as well as overfitting, among other issues
• MobileNet is a streamlined architecture that uses depthwise separable convolutions to construct lightweight deep convolutional neural networks and provides an efficient model for mobile and embedded vision applications
• EfficientNet is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a compound coefficient.
• YOLO is an algorithm that uses neural networks to provide real-time object detection. This algorithm is popular because of its speed and accuracy. It has been used in various applications to detect traffic signals, people, parking meters, and animals
• Types of CNN:
• AlexNet: For image classification, as the first CNN neural network to win the ImageNet Challenge in 2012, AlexNet consists of five convolution layers and three fully connected layers. these are some of the features used that are new approaches to convolutional neural networks:
• ReLU Nonlinearity. AlexNet uses Rectified Linear Units (ReLU) instead of the tanh function, which was standard at the time. ReLU’s advantage is in training time; a CNN using ReLU was able to reach a 25% error on the CIFAR-10 dataset six times faster than a CNN using tanh.
• Multiple GPUs.  GPUs were still rolling around with 3 gigabytes of memory. This was especially bad because the training set had 1.2 million images. AlexNet allows for multi-GPU training by putting half of the model’s neurons on one GPU and the other half on another GPU. Not only does this mean that a bigger model can be trained, but it also cuts down on the training time.
• Overlapping Pooling. CNNs traditionally “pool” outputs of neighboring groups of neurons with no overlapping. However, when the authors introduced overlap, they saw a reduction in error by about 0.5% and found that models with overlapping pooling generally find it harder to overfit.
• • VGG-16: To achieve higher accuracy, VGG-16 is trained to a deeper structure of 16 layers consisting of 13 convolution layers and three fully connected layers, requiring 138 million weights and 15.5G MACS to classify the image with a size of 224x224.
• It makes the improvement over AlexNet by replacing large kernel-sized filters (11 and 5 in the first and second convolutional layer, respectively) with multiple 3×3 kernel-sized filters one after another. VGG16 was trained for weeks and was using NVIDIA Titan Black GPU’s.
• • GoogleNet: To improve accuracy while reducing the computation of DNN inference, GoogleNet introduces an inception module composed of different sized filters. As a result, GoogleNet achieves a better accuracy performance than VGG-16 while only requiring seven million weights and 1.43G MACS to process the image with the same size.
• It achieved a top-5 error rate of 6.67%! This was very close to human level performance which the organisers of the challenge were now forced to evaluate.
• • ResNet, also known as residual neural network, refers to the idea of ??adding residual learning to the traditional convolutional neural network, which solves the problem of gradient dispersion and accuracy degradation (training set) in deep networks, so that the network can get more and more The deeper, both the accuracy and the speed are controlled..
• Advantages:
• The main advantage of CNN compared to its predecessors is that it automatically detects the important features without any human supervision
• CNN is also computationally efficient
• It uses special convolution and pooling operations and performs parameter sharing. This enables CNN models to run on any device, making them universally attractive
• Very High accuracy in image recognition problems.
• Weight sharing.
• Disadvantages:
• CNN do not encode the position and orientation of object.
• Lack of ability to be spatially invariant to the input data.
• Lots of training data is required
• A recurrent neural network is a type of artificial neural network commonly used in speech recognition and natural language processing.It recognize data's sequential characteristics. Recurrent neural networks produce predictive results in sequential data that other algorithms can't.
• Recurrent neural networks (RNN) are more complex. They save the output of processing nodes and feed the result back into the model (they did not pass the information in one direction only). This is how the model is said to learn to predict the outcome of a layer. Each node in the RNN model acts as a memory cell, continuing the computation and implementation of operations. If the network’s prediction is incorrect, then the system self-learns and continues working towards the correct prediction during backpropagation.
• These deep learning algorithms are commonly used for ordinal or temporal problems, such as language translation, natural language processing (nlp), speech recognition, and image captioning; they are incorporated into popular applications such as Siri, voice search, and Google Translate. Like feedforward and convolutional neural networks (CNNs), recurrent neural networks utilize training data to learn. They are distinguished by their “memory” as they take information from prior inputs to influence the current input and output. While traditional deep neural networks assume that inputs and outputs are independent of each other, the output of recurrent neural networks depend on the prior elements within the sequence. While future events would also be helpful in determining the output of a given sequence, unidirectional recurrent neural networks cannot account for these events in their predictions.
• Basic Architecture of RNN:
•  
• Fig (a)Fig(b)
• Recurrent networks share parameters across each layer of the network. Recurrent neural networks leverage backpropagation through time (BPTT) algorithm to determine the gradients, which is slightly different from traditional backpropagation as it is specific to sequence data. The principles of BPTT are the same as traditional backpropagation, where the model trains itself by calculating errors from its output layer to its input layer. These calculations allow us to adjust and fit the parameters of the model appropriately. BPTT differs from the traditional approach in that BPTT sums errors at each time step whereas feedforward networks do not need to sum errors as they do not share parameters across each layer.
• Through this process, RNNs tend to run into two problems, known as exploding gradients and vanishing gradients. These issues are defined by the size of the gradient, which is the slope of the loss function along the error curve. When the gradient is too small, it continues to become smaller, updating the weight parameters until they become insignificant—i.e. 0. When that occurs, the algorithm is no longer learning. Exploding gradients occur when the gradient is too large, creating an unstable model. In this case, the model weights will grow too large, and they will eventually be represented as NaN. One solution to these issues is to reduce the number of hidden layers within the neural network, eliminating some of the complexity in the RNN model.
• Variant RNN architectures
• Bidirectional recurrent neural networks (BRNN): These are a variant network architecture of RNNs. While unidirectional RNNs can only drawn from previous inputs to make predictions about the current state, bidirectional RNNs pull in future data to improve the accuracy of it. If we return to the example of “feeling under the weather” earlier in this article, the model can better predict that the second word in that phrase is “under” if it knew that the last word in the sequence is “weather.”
• Long short-term memory (LSTM): This is a popular RNN architecture, which was introduced by Sepp Hochreiter and Juergen Schmidhuber as a solution to vanishing gradient problem. In their paper (PDF, 388 KB) (link resides outside IBM), they work to address the problem of long-term dependencies. That is, if the previous state that is influencing the current prediction is not in the recent past, the RNN model may not be able to accurately predict the current state. As an example, let’s say we wanted to predict the italicized words in following, “Alice is allergic to nuts. She can’t eat peanut butter.” The context of a nut allergy can help us anticipate that the food that cannot be eaten contains nuts. However, if that context was a few sentences prior, then it would make it difficult, or even impossible, for the RNN to connect the information. To remedy this, LSTMs have “cells” in the hidden layers of the neural network, which have three gates–an input gate, an output gate, and a forget gate. These gates control the flow of information which is needed to predict the output in the network.  For example, if gender pronouns, such as “she”, was repeated multiple times in prior sentences, you may exclude that from the cell state.
• Gated recurrent units (GRUs): This RNN variant is similar the LSTMs as it also works to address the short-term memory problem of RNN models. Instead of using a “cell state” regulate information, it uses hidden states, and instead of three gates, it has two—a reset gate and an update gate. Similar to the gates within LSTMs, the reset and update gates control how much and which information to retain.
• Types of RNN:
• One-to-one: This is also called Plain Neural networks. It deals with a fixed size of the input to the fixed size of output, where they are independent of previous information/output.
•  
• One-to-Many: It deals with a fixed size of information as input that gives a sequence of data as output.
•  
• Many-to-One: It takes a sequence of information as input and outputs a fixed size of the output.
•  
• Many-to-Many: It takes a Sequence of information as input and processes the recurrently outputs as a Sequence of data.
•  
• Bidirectional Many-to-Many: Synced sequence input and output. Notice that in every case are no pre-specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like.
•  
• Advantages:
• RNN can model a sequence of data so that each sample can be assumed to be dependent on previous ones.
• A recurrent neural network is even used with convolutional layers to extend the active pixel neighborhood.
• Disadvantages:
• Gradient vanishing and exploding problems.
• Training an RNN is a complicated task.
• It could not process very long sequences if it were using tanh or relu like an activation function

The above IT Assignment has been solved by our  IT Assignment  Experts at My Uni Paper. Our Assignment Writing Experts are efficient to provide a fresh solution to this question. We are serving more than 10000+ Students in Australia, UK & US by helping them to score HD in their academics. Our experts are well trained to follow all marking rubrics & referencing style.

Be it a used or new solution, the quality of the work submitted by our assignment experts remains unhampered. You may continue to expect the same or even better quality with the used and new assignment solution files respectively. There’s one thing to be noticed that you could choose one between the two and acquire considered worthy of the highest distinction.