Deep Learning with Keras – Discuss Deep Learning essentially means training an Artificial Neural Network (ANN) with a huge amount of data. In deep learning, the network learns by itself and thus requires humongous data for learning. In this tutorial, you will learn the use of Keras in building deep neural networks. We shall look at the practical examples for teaching.
Loading Model for Predictions To predict the unseen data, you first need to load the trained model into the memory. This is done using the following command − model = load_model (”./models/handwrittendigitrecognition.h5”) Note that we are simply loading the .h5 file into memory. This sets up the entire neural network in memory along with the weights assigned to each layer. Now, to do your predictions on unseen data, load the data, let it be one or more items, into the memory. Preprocess the data to meet the input requirements of our model as what you did on your training and test data above. After preprocessing, feed it to your network. The model will output its prediction.
Deep Learning with Keras – Quick Guide Deep Learning with Keras – Introduction Deep Learning has become a buzzword in recent days in the field of Artificial Intelligence (AI). For many years, we used Machine Learning (ML) for imparting intelligence to machines. In recent days, deep learning has become more popular due to its supremacy in predictions as compared to traditional ML techniques. Deep Learning essentially means training an Artificial Neural Network (ANN) with a huge amount of data. In deep learning, the network learns by itself and thus requires humongous data for learning. While traditional machine learning is essentially a set of algorithms that parse data and learn from it. They then used this learning for making intelligent decisions. Now, coming to Keras, it is a high-level neural networks API that runs on top of TensorFlow – an end-to-end open source machine learning platform. Using Keras, you easily define complex ANN architectures to experiment on your big data. Keras also supports GPU, which becomes essential for processing huge amount of data and developing machine learning models. In this tutorial, you will learn the use of Keras in building deep neural networks. We shall look at the practical examples for teaching. The problem at hand is recognizing handwritten digits using a neural network that is trained with deep learning. Just to get you more excited in deep learning, below is a screenshot of Google trends on deep learning here − As you can see from the diagram, the interest in deep learning is steadily growing over the last several years. There are many areas such as computer vision, natural language processing, speech recognition, bioinformatics, drug design, and so on, where the deep learning has been successfully applied. This tutorial will get you quickly started on deep learning. So keep reading! Deep Learning with Keras – Deep Learning As said in the introduction, deep learning is a process of training an artificial neural network with a huge amount of data. Once trained, the network will be able to give us the predictions on unseen data. Before I go further in explaining what deep learning is, let us quickly go through some terms used in training a neural network. Neural Networks The idea of artificial neural network was derived from neural networks in our brain. A typical neural network consists of three layers — input, output and hidden layer as shown in the picture below. This is also called a shallow neural network, as it contains only one hidden layer. You add more hidden layers in the above architecture to create a more complex architecture. Deep Networks The following diagram shows a deep network consisting of four hidden layers, an input layer and an output layer. As the number of hidden layers are added to the network, its training becomes more complex in terms of required resources and the time it takes to fully train the network. Network Training After you define the network architecture, you train it for doing certain kinds of predictions. Training a network is a process of finding the proper weights for each link in the network. During training, the data flows from Input to Output layers through various hidden layers. As the data always moves in one direction from input to output, we call this network as Feed-forward Network and we call the data propagation as Forward Propagation. Activation Function At each layer, we calculate the weighted sum of inputs and feed it to an Activation function. The activation function brings nonlinearity to the network. It is simply some mathematical function that discretizes the output. Some of the most commonly used activations functions are sigmoid, hyperbolic, tangent (tanh), ReLU and Softmax. Backpropagation Backpropagation is an algorithm for supervised learning. In Backpropagation, the errors propagate backwards from the output to the input layer. Given an error function, we calculate the gradient of the error function with respect to the weights assigned at each connection. The calculation of the gradient proceeds backwards through the network. The gradient of the final layer of weights is calculated first and the gradient of the first layer of weights is calculated last. At each layer, the partial computations of the gradient are reused in the computation of the gradient for the previous layer. This is called Gradient Descent. In this project-based tutorial you will define a feed-forward deep neural network and train it with backpropagation and gradient descent techniques. Luckily, Keras provides us all high level APIs for defining network architecture and training it using gradient descent. Next, you will learn how to do this in Keras. Handwritten Digit Recognition System In this mini project, you will apply the techniques described earlier. You will create a deep learning neural network that will be trained for recognizing handwritten digits. In any machine learning project, the first challenge is collecting the data. Especially, for deep learning networks, you need humongous data. Fortunately, for the problem that we are trying to solve, somebody has already created a dataset for training. This is called mnist, which is available as a part of Keras libraries. The dataset consists of several 28×28 pixel images of handwritten digits. You will train your model on the major portion of this dataset and the rest of the data would be used for validating your trained model. Project Description The mnist dataset consists of 70000 images of handwritten digits. A few sample images are reproduced here for your reference Each image is of size 28 x 28 pixels making it a total of 768 pixels of various gray scale levels. Most of the pixels tend towards black shade while only few of them are towards white. We will put the distribution of these pixels in an array or a vector. For example, the distribution of pixels for a typical image of digits 4 and 5 is shown in the figure below. Each image is of size 28 x 28 pixels making it a total of 768 pixels of various gray scale levels.
Deep Learning with Keras – Conclusion Keras provides a high level API for creating deep neural network. In this tutorial, you learned to create a deep neural network that was trained for finding the digits in handwritten text. A multi-layer network was created for this purpose. Keras allows you to define an activation function of your choice at each layer. Using gradient descent, the network was trained on the training data. The accuracy of the trained network in predicting the unseen data was tested on the test data. You learned to plot the accuracy and error metrics. After the network is fully trained, you saved the network model for future use.
Deep Learning with Keras – Useful Resources The following resources contain additional information on Deep Learning with Keras. Please use them to get more in-depth knowledge on this. Useful Links on Deep Learning with Keras − Deep Learning with Keras, its history and various other terms has been explained in simple language. Useful Books on Deep Learning with Keras To enlist your site on this page, please drop an email to [email protected]
Deep Learning with Keras – Saving Model We will save the trained model in our local drive in the models folder in our current working directory. To save the model, run the following code − directory = “./models/” name = ”handwrittendigitrecognition.h5” path = os.path.join(save_dir, name) model.save(path) print(”Saved trained model at %s ” % path) The output after running the code is shown below − Now, as you have saved a trained model, you may use it later on for processing your unknown data.
Deep Learning with Keras – Importing Libraries We first import the various libraries required by the code in our project. Array Handling and Plotting As typical, we use numpy for array handling and matplotlib for plotting. These libraries are imported in our project using the following import statements import numpy as np import matplotlib import matplotlib.pyplot as plot Suppressing Warnings As both Tensorflow and Keras keep on revising, if you do not sync their appropriate versions in the project, at runtime you would see plenty of warning errors. As they distract your attention from learning, we shall be suppressing all the warnings in this project. This is done with the following lines of code − # silent all warnings import os os.environ[”TF_CPP_MIN_LOG_LEVEL”]=”3” import warnings warnings.filterwarnings(”ignore”) from tensorflow.python.util import deprecation deprecation._PRINT_DEPRECATION_WARNINGS = False Keras We use Keras libraries to import dataset. We will use the mnist dataset for handwritten digits. We import the required package using the following statement from keras.datasets import mnist We will be defining our deep learning neural network using Keras packages. We import the Sequential, Dense, Dropout and Activation packages for defining the network architecture. We use load_model package for saving and retrieving our model. We also use np_utils for a few utilities that we need in our project. These imports are done with the following program statements − from keras.models import Sequential, load_model from keras.layers.core import Dense, Dropout, Activation from keras.utils import np_utils When you run this code, you will see a message on the console that says that Keras uses TensorFlow at the backend. The screenshot at this stage is shown here − Now, as we have all the imports required by our project, we will proceed to define the architecture for our Deep Learning network.
Deep Learning with Keras – Training the Model The model training is done in one single method call called fit that takes few parameters as seen in the code below − history = model.fit(X_train, Y_train, batch_size=128, epochs=20, verbose=2, validation_data=(X_test, Y_test))) The first two parameters to the fit method specify the features and the output of the training dataset. The epochs is set to 20; we assume that the training will converge in max 20 epochs – the iterations. The trained model is validated on the test data as specified in the last parameter. The partial output of running the above command is shown here − Train on 60000 samples, validate on 10000 samples Epoch 1/20 – 9s – loss: 0.2488 – acc: 0.9252 – val_loss: 0.1059 – val_acc: 0.9665 Epoch 2/20 – 9s – loss: 0.1004 – acc: 0.9688 – val_loss: 0.0850 – val_acc: 0.9715 Epoch 3/20 – 9s – loss: 0.0723 – acc: 0.9773 – val_loss: 0.0717 – val_acc: 0.9765 Epoch 4/20 – 9s – loss: 0.0532 – acc: 0.9826 – val_loss: 0.0665 – val_acc: 0.9795 Epoch 5/20 – 9s – loss: 0.0457 – acc: 0.9856 – val_loss: 0.0695 – val_acc: 0.9792 The screenshot of the output is given below for your quick reference − Now, as the model is trained on our training data, we will evaluate its performance.
Deep Learning with Keras – Compiling the Model The compilation is performed using one single method call called compile. model.compile(loss=”categorical_crossentropy”, metrics=[”accuracy”], optimizer=”adam”) The compile method requires several parameters. The loss parameter is specified to have type ”categorical_crossentropy”. The metrics parameter is set to ”accuracy” and finally we use the adam optimizer for training the network. The output at this stage is shown below − Now, we are ready to feed in the data to our network. Loading Data As said earlier, we will use the mnist dataset provided by Keras. When we load the data into our system, we will split it in the training and test data. The data is loaded by calling the load_data method as follows − (X_train, y_train), (X_test, y_test) = mnist.load_data() The output at this stage looks like the following − Now, we shall learn the structure of the loaded dataset. The data that is provided to us are the graphic images of size 28 x 28 pixels, each containing a single digit between 0 and 9. We will display the first ten images on the console. The code for doing so is given below − # printing first 10 images for i in range(10): plot.subplot(3,5,i+1) plot.tight_layout() plot.imshow(X_train[i], cmap=”gray”, interpolation=”none”) plot.title(“Digit: {}”.format(y_train[i])) plot.xticks([]) plot.yticks([]) In an iterative loop of 10 counts, we create a subplot on each iteration and show an image from X_train vector in it. We title each image from the corresponding y_train vector. Note that the y_train vector contains the actual values for the corresponding image in X_train vector. We remove the x and y axes markings by calling the two methods xticks and yticks with null argument. When you run the code, you would see the following output − Next, we will prepare data for feeding it into our network.
Evaluating Model Performance To evaluate the model performance, we call evaluate method as follows − loss_and_metrics = model.evaluate(X_test, Y_test, verbose=2) To evaluate the model performance, we call evaluate method as follows − loss_and_metrics = model.evaluate(X_test, Y_test, verbose=2) We will print the loss and accuracy using the following two statements − print(“Test Loss”, loss_and_metrics[0]) print(“Test Accuracy”, loss_and_metrics[1]) When you run the above statements, you would see the following output − Test Loss 0.08041584826191042 Test Accuracy 0.9837 This shows a test accuracy of 98%, which should be acceptable to us. What it means to us that in 2% of the cases, the handwritten digits would not be classified correctly. We will also plot accuracy and loss metrics to see how the model performs on the test data. Plotting Accuracy Metrics We use the recorded history during our training to get a plot of accuracy metrics. The following code will plot the accuracy on each epoch. We pick up the training data accuracy (“acc”) and the validation data accuracy (“val_acc”) for plotting. plot.subplot(2,1,1) plot.plot(history.history[”acc”]) plot.plot(history.history[”val_acc”]) plot.title(”model accuracy”) plot.ylabel(”accuracy”) plot.xlabel(”epoch”) plot.legend([”train”, ”test”], loc=”lower right”) The output plot is shown below − As you can see in the diagram, the accuracy increases rapidly in the first two epochs, indicating that the network is learning fast. Afterwards, the curve flattens indicating that not too many epochs are required to train the model further. Generally, if the training data accuracy (“acc”) keeps improving while the validation data accuracy (“val_acc”) gets worse, you are encountering overfitting. It indicates that the model is starting to memorize the data. We will also plot the loss metrics to check our model’s performance. Plotting Loss Metrics Again, we plot the loss on both the training (“loss”) and test (“val_loss”) data. This is done using the following code − plot.subplot(2,1,2) plot.plot(history.history[”loss”]) plot.plot(history.history[”val_loss”]) plot.title(”model loss”) plot.ylabel(”loss”) plot.xlabel(”epoch”) plot.legend([”train”, ”test”], loc=”upper right”) The output of this code is shown below − As you can see in the diagram, the loss on the training set decreases rapidly for the first two epochs. For the test set, the loss does not decrease at the same rate as the training set, but remains almost flat for multiple epochs. This means our model is generalizing well to unseen data. Now, we will use our trained model to predict the digits in our test data.